# Using design of experiments with discrete event simulation in operational research: a review

Amalia Gjerloev, Luca Grieco, Guillaume Lamé, Sonya Crowe

<figure><img src="image_1.png" /></figure>

## To cite this version:

Amalia Gjerloev, Luca Grieco, Guillaume Lamé, Sonya Crowe. Using design of experiments with discrete event simulation in operational research: a review. Journal of Simulation, In press, pp.1-17. 10.1080/1747778.2025.2527151. hal-05152363

HAL Id: hal-05152363

https://hal.science/hal-05152363v1

Submitted on 11 Jul 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Using design of experiments with discrete event simulation in operational research: a review

Amalia Gjerloev, Luca Grieco, Guillaume Lamé & Sonya Crowe

**To cite this article:** Amalia Gjerloev, Luca Grieco, Guillaume Lamé & Sonya Crowe (04 Jul 2025): Using design of experiments with discrete event simulation in operational research: a review, Journal of Simulation, DOI: 10.1080/17477778.2025.2527151

**To link to this article:** https://doi.org/10.1080/17477778.2025.2527151

<figure><img src="image_3.png" /></figure>

© 2025 The Author(s). Published by Informa UK Limited, trading as Taylor & Francis Group.

<figure><img src="image_4.png" /></figure>

Published online: 04 Jul 2025.

<figure><img src="image_5.png" /></figure>

Submit your article to this journal

<figure><img src="image_6.png" /></figure>

<figure><img src="image_7.png" /></figure>

Article views: 94

<figure><img src="image_8.png" /></figure>

View related articles

<figure><img src="image_9.png" /></figure>

<figure><img src="image_10.png" /></figure>

View Crossmark data

<figure><img src="image_11.png" /></figure>

CrossMark
<figure><img src="image_12.png" /></figure>

<figure><img src="image_13.png" /></figure>

REVIEW ARTICLE

<figure><img src="image_14.png" /></figure>

OPEN ACCESS

<figure><img src="image_15.png" /></figure>

Check for updates

# Using design of experiments with discrete event simulation in operational research: a review

Amalia Gjerloev IDª, Luca Grieco IDª, Guillaume Lamé IDᵇ and Sonya Crowe IDª

ªClinical Operational Research Unit, Department of Mathematics, University College London, London, UK; ᵇLaboratoire de Génie Industriel, CentraleSupélec, Paris-Saclay University, Gif-sur-Yvette, France

## ABSTRACT

Design of experiments (DoE) is a branch of applied statistical methods, sometimes referred to as experimental design, that aims to efficiently assess the impact of input parameters on a system's responses. DoE covers a range of techniques that can help modellers and end-users better understand the system under different scenarios and support the identification of parameter configurations that lead to superior performance. While potentially useful for a better understanding of the interplay between different parameters in discrete-event simulation (DES) models, DoE is rarely used in such contexts. We conducted a review of the literature and identified 86 papers that used DoE techniques to analyse DES models. We found that the majority of these articles applied simple DoE techniques, namely full-factorial designs, to case studies. However, more advanced methods have also been combined with DES, both in methodological and applied articles. We present a case study that illustrates how DoE can be incorporated with a DES model in a healthcare context and enhance our operational decision-making. We review the benefits of using different experimental designs and metamodels in the context of a patient flow model through an enhanced care unit.

## ARTICLE HISTORY

Received 26 August 2024
Accepted 26 June 2025

## KEYWORDS

Simulation; design of experiments; discrete event simulation; metamodels

## 1. Introduction

Design of experiments (DoE) is a set of applied statistics methods used to learn how certain parameters affect the behaviour of a system. DoE methods support the systematic exploration of parameter values, including their interactions. This approach is much more effective than running a select set of scenarios or the “one-factor-at-a-time” approach, which examines one parameter while keeping others constant (Law, 2017). By exploring combinations of parameter values simultaneously, DoE systematically reduces the total number of experiments required, making it advantageous when exploring the interactions between parameters and dealing with complex systems. Using DoE results, experimenters can also build metamodels of their system of interest, and use these metamodels to identify optimal and robust/resilient parameter configurations (J. Kleijnen et al., 2005).

One potentially beneficial application of DoE is within analyses that explore the potential impact of system changes using discrete-event-simulation (DES). In such studies, DoE offers a systematic and efficient approach for exploring the influence of underlying parameters. However, despite this potential, there is a lack of evidence about how DoE techniques are being used in this context.

In order to demystify the use of DoE for DES modellers and to encourage its application, we make two contributions. First, we assess the current state of DoE in DES practice in academia. Second, we illustrate how DoE techniques, both basic and more advanced, can be combined with DES using existing packages. We present a healthcare case study that shows how different DoE techniques perform on DES models. We hope that this can be of interest both to newcomers to DoE and to people who already have a basic command but would like to explore other techniques.

The next section provides background knowledge on DoE and how it has been used with DES. In Section 3, we present the methodology we used to conduct the literature review, Section 4 details the results and presents a narrative synthesis of the included studies, and in Section 5, we present a case study illustrating the benefits of using DoE techniques with DES models. Finally, we summarise our findings and highlight key insights in Section 6.

## 2. Background

### 2.1. Design of experiments

We provide a brief overview of the terminology and range of approaches to DoE, with readers referred to other published DoE reviews and tutorials (J. Kleijnen

<figure><img src="image_16.png" /></figure>

<figure><img src="image_17.png" /></figure>
<figure><img src="image_18.png" /></figure>

et al., 2005; Law, 2013, 2017; Sanchez et al., 2012, 2020) for further details.

In DoE, input parameters that the experimenter manipulates are referred to as “factors”. Each factor can be continuous or discrete, with each value defined as a “level” of the factor. Manipulating the levels of each factor, experimenters measure the “responses” (outputs or performance measures) of the system (Montgomery, 2017). A given combination of levels for all the factors is an experimental design or scenario (Montgomery, 2017). Each experimental design can be run one or several times.

The simplest approach to DoE is the full factorial design, where each factor has a low and a high level, and all combinations of these two levels are studied. With k factors, this leads to 2^k experiments. Because of this combinatorial explosion, and the cost and time needed to run experiments, more efficient methods have been developed to allow experimenters to learn about the system they study with fewer experiments (J. Kleijnen et al., 2005). For example, fractional factorial designs remove selected combinations from a full factorial design, to reduce the number of scenarios that need to be tested (Law, 2017).

Metamodels, mathematical expressions that model the response variable as a function of relevant input variables, are used to gain a deeper understanding of how factors impact the response variable, predict how the system may behave under system configurations not yet tested, and find input values that optimise the response. First-order regression models only need a simple full-factorial design for calculating all the coefficients. Second-order regression models are similar to linear regression models, however they consider higher order terms, such as the interaction between factors. For these, a more complicated design is needed, e.g., central composite designs (CCD), Box-Behnken, three-level full factorial design (a.k.a. 3^k designs), or Latin Hypercube Sampling (LHS) designs (Law, 2017). CCDs start from a full- or fractional-factorial design, to which centerpoints are added on each facet (Box & Wilson, 1951). Box-Behnken designs require three levels per factor, but only a subset of the combinations are run (Box & Behnken, 1960). LHS designs are built so that, with k factors, each divided in n levels, only one point is sampled at each level of each factor.

Modellers can also choose to implement higher order metamodels, smooth metamodels (e.g., Kriging metamodels), or non-smooth metamodels (e.g., neural network metamodels). Kriging metamodels are based on interpolation: the output is a weighted linear combination of all output values already observed, with weights based on the distances between the input for which the output is to be predicted and the inputs already simulated (J. P. Kleijnen, 2009; J. P. C. Kleijnen, 2005). Neural network metamodels

are artificial intelligence models that use algorithms and neural network structures, designed to mimic the structure of the human brain (Fonseca et al., 2003). These models are capable of learning patterns and relationships in data, enabling them to predict outputs.

Optimisation can be performed on the metamodels to find a set of near-optimal solutions. Simulation-optimisation with DoE approaches involve combining techniques, namely a design of experiment and a stochastic simulation model to create a dataset sampling the relationship between the input and output of interest; a learning procedure such as linear regression or a more sophisticated machine learning technique (e.g., artificial neural networks) to identify a metamodel of the input/output relationship; an optimisation or heuristic procedure to explore the feasible combinations of parameters (using the metamodel) and identify a (sub-)optimal parameter configuration (J. P. Kleijnen & Kleijnen, 2015).

## 2.2. DoE in discrete-event simulation

DoE can be coupled with computer simulation. JPC Kleijnen, and RG Sargent (2000) cover metamodeling for simulation. J. P. C. Kleijnen (2009) focuses on Kriging metamodeling. J. P. C. Kleijnen (2017) looks at regression and Kriging metamodeling. J. P. C. Kleijnen (2005) examines DoE for sensitivity analysis. J. Kleijnen et al. (2005) outline how simulation can be used with DoE, review how different designs may be appropriate for different modelling situations, and propose a “design toolkit” to help modellers choose experimental designs. The choice of design reflects the number of factors, anticipated complexity of the system, and the type of metamodel/ response surface.

However, none of these reviews are specific to DES. Even if DoE is a staple in the methodological toolbox of industrial and quality engineers, popular textbooks on DES barely touch upon DoE and do not go further than classic full factorial and fractional factorial designs (Altiok & Melamed, 2007; Banks et al., 2005; Law, 2013; Robinson, 2004; Rossetti, 2016). These designs take into consideration few factors, typically fewer than 10. Yet, one of the interests of simulation is precisely to be able to assess the influence of various factors, as simulation allows testing of many more configurations than physical experiments (J. Kleijnen et al., 2005). J. P. C. Kleijnen (2008a) is the key reference on DoE methodology for DES. J. P. C. Kleijnen (2008b) discusses statistical issues with simple regression metamodels for DES.

Despite the value of DoE in gaining insights about systems and the relatively low cost of conducting experiments with DES models, DoE within DES studies is far from ubiquitous. In
<figure><img src="image_19.png" /><figcaption>**Table 1.** Data extraction parameters.</figcaption></figure>

<table><thead><tr><td>Field</td><td>Description</td></tr></thead><tbody><tr><td>Article type</td><td>Did the article apply DoE to a problem, or was it methodological aiming to advance DoE and its application to simulation?</td></tr><tr><td>Case study</td><td>Did the paper present a case study or apply DoE to a real world problem</td></tr><tr><td>DoE methodology</td><td>What steps did the authors follow in their analysis?</td></tr><tr><td>Type of DoE methods</td><td>What designs did the authors use?</td></tr><tr><td>Justification of method use</td><td>Did the authors provide a reason for applying the techniques?</td></tr><tr><td>Number of factors considered</td><td>How many factors were included in the design?</td></tr><tr><td>Response surface or other metamodel</td><td>Did the paper apply a metamodel or response surface, and if so what model did they use?</td></tr></tbody></table>

a questionnaire study, Hollocks (2001) found that only a quarter of modellers used DoE to select what scenarios to simulate. More recently, Hoad et al. (2015) surveyed 302 simulation academics and practitioners and found that 50% of them had practical experience of DoE with DES (2015). J. V. Amaral et al. (2022) reviewed metamodeling-based simulation optimisation and identified 41 articles that combined DES and metamodeling.

Data farming is an another area that utilises DES and DoE to “grow data” purposefully with the intention of gaining further insights into a system. Unlike data mining, which extracts valuable information from existing big data, data farming focuses on generating new data through simulation models and experimental designs. The experimental designs take a “big view of factors, response features, and analysis flexibility” (Sanchez, 2020). Some designs such as Nearly Orthogonal Latin Hypercube (NOLH), Saturated Nearly Orthogonal Latin Hypercube (S-NOLH), and resolution V fractional factorial designs are therefore well suited for data farming due to their ability to handle a large number of factors with efficiency and orthogonality. Data farming can be useful for training neural network metamodels and automating the process of data generation, thereby reducing analysts’ time and effort. For more information, readers are encouraged to consult reviews and papers such as Sanchez (2017; 2020), Sanchez (2018), and Lechler et al. (2021). In addition, readers are encouraged to refer to the paper by Greasley and Edwards (2021), who provide a review of DES simulation models integrated with big data analytics techniques, including data farming, and propose a framework for incorporating these approaches into a DES methodology.

Historically, one of the issues in combining DoE with DES might have been that DES is often performed on proprietary software, with poor interfaces to programming languages, making it tedious to specify experimental designs with dozens of scenarios. However, simulations written directly in code can be coupled with DoE packages, meaning that with the increasing development of open-source DES in R or Python, DES can now be easily combined with DoE packages in these languages.

# 3. Methodology

## 3.1. Systematic literature review

We searched for relevant articles on the Web of Science, using the following query:

“(design of experiment”[topic] or “doe”[topic] or “sensitivity analysis”[topic]) AND “discrete event simulation”[topic]

The search was run on February 13, 2023, and re-run on May 21, 2025. We included only articles that discussed discrete-event simulation and design of experiments. Other types of simulation were not included. We included both journal and conference articles. We set no restriction on publication dates but only included articles written in English.

Each article was screened by one of the co-authors (single screening). We proceeded in two stages: first, on abstracts and titles; second, on full-texts. When unsure about an article, we discussed it collectively to decide whether to include it or not. Once we obtained a set of articles, each of the articles was assigned to one of the co-authors to extract information (Table 1).

In order to better visualise the methods used in the articles, we also mapped included studies on a framework proposed by Kleijnen et al. (2005) that combines the number of factors with the type of metamodel used.

## 3.2. Case study

To illustrate the application of different DoE techniques on a DES model, we present a case study in healthcare. The DES model is adapted from work conducted by some co-authors with a UK hospital. We show how DoE approaches combined with meta-modelling can help better understand the system than basic, one-parameter-at-a-time analysis, however, with the risk of obtaining misleading insights if the level of approximation is not chosen appropriately. This example also shows how DES and DoE can be integrated easily using existing packages in R. The code to generate the analysis in this case study can be found on GitHub (Grieco, 2024).
<figure><img src="image_20.png" /></figure>

# 4. Results

## 4.1. Study characteristics

Our search query returned 382 articles. After abstract and full-text screening, we included 86 articles for review (Figure 1). A database of the papers included in this review can be found in the additional material. Full and fractional factorial experimental designs were popular, with more than half of the articles using a full factorial design (Table 2). Less than half of the articles

included a metamodel. Of the 29 metamodels described, 19 (68%) were polynomial regressions. Kriging and neural networks were much less represented, and one article used penalised regression splines.

The most common sector among articles was manufacturing processes/service and supply chains, with 29 articles falling in this category. Nine articles were purely theoretical, which was the second most common category, and seven articles were healthcare

<figure><img src="image_21.png" /><figcaption>Figure 1. PRISMA flowchart illustrating the number of papers that were excluded at each step of the data extraction.</figcaption></figure>

<table><caption>Table 2. Characteristics of included articles.</caption><thead><tr><th>Type of article</th><th>Number of articles</th><th>% of articles</th></tr></thead><tbody><tr><td>Application</td><td>55</td><td>64%</td></tr><tr><td>Methodological and reviews</td><td>31</td><td>36%</td></tr><tr><td>incl. methodological articles with a toy model</td><td>23</td><td>27%</td></tr><tr><td><strong>Number of factors included*</strong></td><td></td><td></td></tr><tr><td>Median [interquartile range]</td><td>5 [4; 6]</td><td></td></tr><tr><td>Average</td><td>9.8</td><td></td></tr><tr><td>Min - Max</td><td>2–200</td><td></td></tr><tr><td><strong>Type of sampling**</strong></td><td></td><td></td></tr><tr><td>Full factorial</td><td>50</td><td>58%</td></tr><tr><td>Orthogonal arrays and Taguchi designs</td><td>11</td><td>13%</td></tr><tr><td>Fractional factorial</td><td>7</td><td>8%</td></tr><tr><td>LHS</td><td>4</td><td>5%</td></tr><tr><td>CCD</td><td>4</td><td>5%</td></tr><tr><td>Sequential experiment designs</td><td>4</td><td>5%</td></tr><tr><td>Box-Behnken</td><td>3</td><td>3%</td></tr><tr><td>NOLH</td><td>2</td><td>2%</td></tr><tr><td>Plackett-Burman</td><td>1</td><td>1%</td></tr><tr><td>D-optimal design</td><td>1</td><td>1%</td></tr><tr><td>Fast Flexible Filling design</td><td>1</td><td>1%</td></tr><tr><td>Uniform design</td><td>1</td><td>1%</td></tr><tr><td>Split-plot designs</td><td>1</td><td>1%</td></tr><tr><td>Three-Level Definitive Screening Design (Jones and Nachtsheim 2011)</td><td>1</td><td>1%</td></tr><tr><td>None or insufficient information</td><td>5</td><td>6%</td></tr><tr><td><strong>Metamodeling approach**</strong></td><td></td><td></td></tr><tr><td>First order regression</td><td>3</td><td>3%</td></tr><tr><td>First order regression with interaction terms</td><td>6</td><td>7%</td></tr><tr><td>Second order regression</td><td>12</td><td>14%</td></tr><tr><td>Kriging</td><td>5</td><td>6%</td></tr><tr><td>Neural network</td><td>5</td><td>6%</td></tr><tr><td>Logistic regression</td><td>1</td><td>1%</td></tr><tr><td>Penalised regression splines</td><td>1</td><td>1%</td></tr><tr><td>Gradient boosted trees</td><td>1</td><td>1%</td></tr><tr><td>None or insufficient information</td><td>53</td><td>62%</td></tr></tbody></table>

<footnote>NOTE: * if multiple designs were described in the article, we counted factors in the largest design, ** some articles included more than one design and/or metamodeling approach</footnote>
<figure><img src="image_22.png" /></figure>

<figure><img src="image_23.png" /><figcaption>**Figure 2.** Included studies mapped to the framework provided by Kleijnen et al. (2005): the number of factors used in each study vs. the response surface complexity of the metamodel.</figcaption></figure>

focused, the third most common category. More than half, 30 (55%), of the articles were focused on real-life applications of DoE techniques and included case studies (see Table 2).

Most articles included only few factors: more than half of the articles had five factors or less, and three quarters of the articles had nine factors or less (Table 2, Figure 2). The number of articles that combine DES and DoE has been developing steadily since 2000 (Figure 3). The proportion of articles that use only a full factorial design and the proportion of articles that include a metamodel remained stable in time, at approximately 58% and 40% for the whole set, respectively (see Table 2 and Figures 4 and 5).

**4.2. Narrative synthesis based on response surface complexity**

Referring back to the design and metamodel classification by Kleijnen et al., two papers did not fall into any categories (Hoad et al., 2015; Özdemirel et al., 1996). These are papers that presented generalisable methodologies or reviewed DoE approaches (e.g Hoad et al. (2015)). Özdemirel et al. (1996) outline a step-by-step guide for modellers looking to design a DoE and simulation-based model. The process, “Design of Experiments for Simulation”, takes modellers through the planning, design, and implementation steps. In the paper by Hoad et al., the authors conducted

<figure><img src="image_24.png" /><figcaption>**Figure 3.** Trends in number of articles and methodological vs applied articles.</figcaption></figure>
<figure><img src="image_25.png" /></figure>

<figure><img src="image_26.png" /><figcaption>**Figure 4.** Cumulative number of articles featuring each category of designs (the same article may include more than one design and thus contribute to more than one category).</figcaption></figure>

<figure><img src="image_27.png" /><figcaption>**Figure 5.** Cumulative number of articles featuring each category of metamodels (the same article may include more than one metamodel and thus contribute to more than one category).</figcaption></figure>

a survey of over 300 modellers to understand how DES and experimentation are used in practice (2015). The authors found that outside of academic research, these methods are rarely used, thereby emphasising the importance of the process designed by Özdemirel et al. (1996).

### 4.2.1. No response surface

Papers did not include any response surface methodology (for example, see Figure 6), of which 34 (68%) papers chose full factorial designs. The rest of the papers used orthogonal arrays, fractional

factorial designs, Nearly Orthogonal Latin Hypercube designs, and Taguchi designs. One paper uses a controlled sequential factorial design for screening important factors (Shen & Wan, 2009). Papers that had 35-factors used orthogonal arrays to minimise the number of experiments (Ilgin & Gupta, 2010, 2011a, 2011b).

$2^k$ designs were the most popular full factorial designs and were used in models with less than nine factors. These papers typically conducted an analysis of variance (ANOVA) following a full factorial design in order to identify significant interactions between
<figure><img src="image_28.png" /></figure>

<figure><img src="image_29.png" /><figcaption>Second Order Response Surface</figcaption></figure>

<figure><img src="image_30.png" /><figcaption>Higher Order Response Surface</figcaption></figure>

<figure><img src="image_31.png" /></figure>

<figure><img src="image_32.png" /><figcaption>**Figure 6.** Example figures illustrating design points for two factors with no response surface (top left), and a first order (top right), second order (bottom left), and higher order (bottom right) response surface. The colour bar indicates the response value.</figcaption></figure>

factors. The results were then plotted in interaction plots and further analysed.

Mourtzis et al. modelled the production line of copper tubes, using the Taguchi method to minimise the number of experiments run (2021). In order to determine significant factors, they began by identifying factors and appropriate levels, selected their objective function (maximising throughput), ran experiments, and finally used analysis of means and ANOVA to determine the statistical significance of each factor. Similiarly, Chackelson et al. used a full factorial design with ANOVA to identify significant factors for processing and scheduling orders, releasing orders, and picking and disposing of articles in warehouses (2013). Baril et al. used Tukey's test in addition to ANOVA to conduct pairwise comparison between effects of different factors in an outpatient orthopaedic clinic (2014).

Shen and Wan proposed a methodology using a controlled sequential factorial design (CSFD) with DES to analyse systems with more than 100 factors (2009). In the paper, they compared CSFD with other sequential screening design using a small-scale toy model, 10 factors, and two large-scale toy models, 200 and 500 factors. They showed that CSFD was more efficient in large-scale cases, requiring fewer simulation replications and less sensitive to changes in variances.

### 4.2.2. First-order response surface

Nine papers included a first-order metamodel or a first-order metamodel with interaction terms (for example, see Figure 6), with two papers including additional higher order metamodels (Chan & Spedding, 2001; Fu et al., 2021). Three papers used a fractional factorial design, two used full factorial designs, two used nearly orthogonal Latin Hypercube designs, one paper used Taguchi orthogonal arrays
<figure><img src="image_33.png" /></figure>

(Tozanlı et al., 2020), and the last used a CCD design with a full factorial design. Fu et al. created both first- and second-order regression models, using the first to find the region of optimal solutions and the second to find the optimal value (2021).

Another paper used a fractional factorial design to screen for important factors and then used an additional design, 2^k^ full factorial, to create the first-order linear regression metamodel (Montevechi et al., 2010). In this paper, they used a linear programming application in Excel to solve an objective function fitted to the four important factors. They applied this methodology to a Brazilian weapon factory production line, looking to minimise the production lead time.

Only two papers used Nearly Orthogonal Latin Hypercube (NOLH) designs, both of which applied first-order regression models (Kang & McDonald, 2010; Marlin & Sohn, 2016). Marlin and Sohn (2016) assessed primary and secondary school policies in Afghanistan, testing four different models: a NOLH model with 11 factors, a seven-factor NOLH where six variables were crossed with a 2^k^ full factorial design, a stacked NOLH design, and a second-order space filling design. They fitted a linear regression model and applied data envelopment analysis to assess the impact on the student and teacher retention rates, graduate numbers, and the student-to-teacher ratio. Kang and McDonald used linear regression models and regression trees, and a NOLH design with 257 design points to model 13 factors affecting military vehicle readiness, enabling factor screening and complex metamodel development (2010).

### 4.2.3. Second-order response surface

Second-order metamodels with quadratic terms were implemented in 11 articles (for example, see Figure 6). Central composite designs (CCDs) are practical for deriving second-order metamodels, because they require fewer derivations than full factorial designs with more than three levels per factor, like 3^k^ designs. However, 47% of papers (seven papers) used full factorial designs with three or more levels per factor and only four papers used CCDs. The other papers employed fractional factorial designs, sequential bifurcation, and Box Behnken designs (BBD). One paper used a screening design by Jones & Nachtsheim (2011) and derived a second-order metamodel (Hill et al., 2015).

A few studies explored practical applications of DoE with DES models. Assad et al. (2015) used a full factorial design to analyse the preventative maintenance interventions for manufacturing systems, deriving cost relationships using response surface methodologies. Similarly, Baesler et al. created a simulation model of a sawmill in Chile and used a full factorial design along with a second-order linear

regression model to find optimal machine settings, achieving a 25% productivity increase (2004).

Hussein et al. (2023) used a Box-Behnken design to fit a second-order metamodel for optimising multi-modal logistics in modular integrated construction projects, thereby reducing project duration, costs, and carbon emissions. Corsini et al. (2022) applied quadratic regression and Pareto multi-objective analysis to identify optimal trade-offs between conflicting objectives in outpatient chemotherapy oncology departments, enabling healthcare managers to evaluate ward configurations and performance impacts. The multi-objective analysis determined a Pareto front, comprising a set of “non-dominated” solutions within the feasible set, where a non-dominated solution implies that no objective function can be improved without compromising another.

Červeňanská et al. (2020) compared response surfaces generated by face-centered and BBD designs for batch production, finding that face-centred designs performed better when it came to predicting responses. Wan and Ankenman (2007) demonstrated that a two-stage controlled fractional factorial method was more efficient than sequential bifurcation for screening important factors, requiring less prior information.

Some studies created both first- and second-order linear regression models. Fu et al. (2021) employed a CCD and a 2^k^ full factorial design to assess various automated guided vehicle requirements (e.g., type, speed, capacity, acceleration), creating quadratic and linear regression models. In 2014, Zhang and Brown used response surface methodologies to minimise police response times and workload variation, employing fractional factorial and Latin Hypercube designs to screen for important factors before constructing a second-order metamodel with a CCD.

### 4.2.4. Higher order response surface

We define higher order response surfaces as smooth or non-smooth response surfaces (for example, see Figure 6), such as neural network models (NNM), Kriging interpolation regression models, or penalised regression spline metamodels. Twelve papers used metamodels in this category. Kriging regression models were implemented in five papers, neural network models were used by five papers, one paper used a Bagging algorithm to train a metamodel (J. V. S. D. Amaral et al., 2025), and the final paper used a spline metamodel.

Four papers used LHS designs to fit Kriging regression models (J. V. S. D. Amaral et al., 2025; Jia et al., 2017; Smew et al., 2013; Zhang & Brown, 2014), with one paper highlighting that LHS designs “cover the input domain uniformly in a relatively small sample size” (Smew et al., 2013). One of these papers used a Kriging metamodel to investigate how districting
<figure><img src="image_34.png" /></figure>

plans affect performance measures for police patrols (Zhang & Brown, 2014). In another study, the authors used a neural network metamodel to analyse fibrous materials (Cherkassky, 2011). Five different characteristics of the fibres were used in the model to estimate two outputs: the thickness and the number of fibres in a cross-section.

Nasiri et al. used a full-factorial design with a DES model for open shop scheduling, testing artificial neural network approaches to identify the best metamodel. They applied Data Envelopment Analysis to determine dispatching rules that minimised job waiting times (2017). Similarly, Chan and Spedding used neural network metamodels with a grid-search algorithm to optimise manufacturing systems that experience errors or defects, employing a fractional factorial design to model defect rates. Response surfaces were created and used to identify the optimal defective rates for each machine and calculate the system's total yield percentage and minimum cost per unit (2001).

Yang illustrated how experimental design integrates with neural network metamodels for single/multi-product manufacturing, proposing a feedforward structure with sigmoid activation functions (2010). Meanwhile, Barton presented strategies for conducting simulation optimisation with metamodels, highlighting that linear regression metamodels are often used for local optimisations, while Kriging and neural networks excel in global optimisation with space-filling designs like Latin Hypercubes (2009). They noted that genetic algorithms can be used for unconstrained optimisations with neural networks, with Kriging accommodating both constrained and unconstrained optimisations.

**5. Case study**

In this section, we present an application of DoE with a DES model. The aim of this case study is to use a practical example to give a brief account of the challenges and potential gains of applying DoE and meta-modelling with different levels of sophistication to support decision-making.

Some UK hospitals have an Enhanced Care Unit (ECU) that treats patients who need elevated levels of care such as cardiac monitoring, respiratory support or continuous drugs, but not full intensive care. Patients who might otherwise have been transferred from the Emergency Department (ED) to the Intensive Therapy Unit (ITU) are instead admitted to ECU if their needs can be met there, freeing up capacity in ITU for other patients. These patients might need ITU later on if their condition worsens, or they might be directly transferred elsewhere (ward, home, etc.) without ever being admitted to ITU. Patients might also “step down” from ITU to ECU if they no longer require intensive care but their conditions are not good enough for a transfer to a ward.

Figure 7 shows the possible patient trajectories through ECU and/or ITU. The proportions of patients that follow each trajectory would be determined by clinical thresholds regarding the suitability of ECU versus ITU care and the case mix of patients. We do not consider these clinical algorithms explicitly here; instead, we explore the impact of different proportions that would result from decisions about clinical thresholds or changes in patient case mix.

We consider a baseline case where a hospital's clinical thresholds and patient case mix are such that the flow of incoming patients into the ECU/ITU system is split into the trajectories in the proportions shown in Figure 7. Furthermore, we set the following additional baseline assumptions:

(1) Average overall patient arrival rate: 5.4 patients/day;

(2) ECU capacity: 13 beds;

(3) ITU capacity: 30 beds;

(4) Service times are assumed to be log-normally distributed with mean $τ$ days in the unlogged scale as specified in Figure 7 and standard deviation in the unlogged scale equal to $0.1τ$.

We analyse the effects of lowering the clinical threshold for promoting ECU over ITU use, in order to free up the more expensive ITU capacity. In other words, we consider the impact on performance metrics of increasing the proportions of patients flowing through Trajectories 1, 3, 4, 5 and lowering the proportion of patients in Trajectory 2. Additionally, we consider moving beds between ECU and ITU (total number of beds being fixed) to balance patient flow within the system.

In this example of analysis, we explore how insights regarding the above decisions (i.e., patient split between trajectories and bed distribution) would change with different levels of sophistication of the chosen sensitivity analysis approach. In particular, we considered the following four approaches of increased sophistication to inform the above decisions. For a consistent comparison of the three approaches, we structured them such that we execute the same number of simulation runs in each case.

(1) “What-if” analysis. We conducted a typical scenario analysis where we tested the effects on relevant outputs of combining: (i) different proportions of additional 14 patients allowed in Trajectories 1, 3, 4, 5, and (ii) different numbers of beds in ECU (with corresponding change in ITU beds to keep the total count fixed). We tested a few reasonable values for the corresponding parameters and visualised the outputs of interest in all the scenarios tested. We identified what would be a solution chosen by the decision maker.
<figure><img src="image_35.png" /></figure>

<figure><img src="image_36.png" /><figcaption>**Figure 7.** Patient trajectories. Our baseline assumption is that 24% of the incoming patients stay in the ECU only (trajectory 1 - patients who would not normally need ITU at all but still contribute to demand for ECU beds), 70% stay in the ITU only (trajectory 2), and the remaining 6% would be equally split between trajectories 3, 4, 5. We also report the average service time $τ$ (in days) that we used in our simulations for each step in the trajectories.</figcaption></figure>

(2) Full Factorial Design (FFD) with first-order meta-model. To assess the robustness of the above decision with respect to additional exogenous parameters of the models, we exploit DoE approaches and response surfaces, namely sampling model output based on a full-factorial design with three levels per factor (3^k^ design) and fitting a first-order meta-model. In particular, we considered two additional factors: average overall patient arrival rate; split of service time between ECU and ITU in Trajectories 3, 4, 5 (how much of the original ITU length of need is now spent in each of the ECU stays in a trajectory, assuming that the average total length of need for the overall patient journey is fixed across all trajectories).

(3) FFD with second-order meta-model. Same as item (2), but with a second-order meta-model fitted to the produced data.

(4) LHS with second-order meta-model. Same as item (2), but with an experimental design based on Latin Hypercube Sampling and a second-order meta-model fitted to the produced data.

We compared simulation scenarios in terms of expected patient waiting time (expressed in hours) for ECU, ITU, and overall, averaged across 100 simulation runs.

## 5.1. What-if analysis results

We tested 81 scenarios covering all combinations of the following varying parameter values:

* $α \in \{0\%, 5\%, 10\%, 15\%, 20\%, 25\%, 30\%, 35\%, 40\%\}$ denoting the additional proportion of overall arrivals that, instead of following Trajectory 2, follow Trajectories 1, 3, 4, 5 (equally split);

* $B_{ECU} \in \{9, 10, 11, 12, 13, 14, 15, 16, 17\}$ denoting the number of beds deployed in ECU, assuming that the total number of beds across ECU and ITU remains constant.

Results are summarised in Figure 8. Increasing values of $α$ are associated with increased waiting times in ECU and decreased waiting times in ITU, therefore pushing the decision-maker towards an increase in the number of ECU beds.

Based on visual inspection of Figure 8, we identified three decisions that would likely be made for three selected scenario in terms of $α$, determining capacity levels that would give the shortest expected patient waiting time in the corresponding scenario:

* $B_{ECU} = 12$ for $α = 0\%$

* $B_{ECU} = 14$ for $α = 15\%$

* $B_{ECU} = 16$ for $α = 30\%$

## 5.2. FFD with first-order meta-model

We tested 81 scenarios covering all combinations of the following varying parameter values:

* $α \in \{0\%, 20\%, 40\%\}$;

* $B_{ECU} \in \{9, 13, 17\}$;
<figure><img src="image_37.png" /></figure>

<figure><img src="image_38.png" /><figcaption>**Figure 8.** What-if analysis. contour plots summarise expected patient waiting times (in hours) with changing values of $\alpha$ and ECU beds ($B_{ECU}$), for ECU, ITU and overall, respectively. Asterisks denote decisions made by the decision maker on how many beds to have in ECU, with three different, arbitrarily selected values of $\alpha$: $\{0, 0.15, 0.3\}$.</figcaption></figure>

* $\lambda \in \{2.4, 5.4, 8.4\}$;

* $\beta \in \{10\%, 25\%, 40\%\}$.

where $\lambda$ is the expected arrival rate (we used a range of $\pm 3$ of the baseline arrival rate) and $\beta$ is the proportion of ITU service time that is instead ECU service time in Trajectories 3, 4, 5.

Results are summarised in Figure 9. For the sake of brevity, we only report our analysis on robustness of the decision with respect to $\lambda$. For a chosen value of $\alpha$, we can explore how the decisions defined in the “what-if” analysis would change if there were changes in the overall average arrival rate. Results seem to suggest that, no matter the value of $\alpha$, the choice made with the “what-if” analysis would be improved by increasing the number of beds in ECU up to the maximum allowed value. While this would need further exploration to verify that the corresponding waiting times in the ECU and in the ITU would be reasonable, we can observe how the

waiting time predictions made using a first-order meta-model deviate quite a lot from the values obtained in the what-if analysis (for the baseline arrival rate). This suggests that a first-order model is not enough to capture the complexity of the relationship between number of beds and waiting time. In this case study, this model might lead to misleading conclusions.

### 5.3. FFD and LHS with second-order metamodel

In this sub-section, we report results for both items J. V. S. D. Amaral et al. (2025) and Ankenman et al. (2010). In both cases, we fitted the produced data using a second-order meta-model. For the FFD, we used the same dataset produced for the first-order meta-model fitting. For the LHS design, we used 81 combinations of the parameters of interest, randomly sampled (uniformly) over the following ranges:

<figure><img src="image_39.png" /><figcaption>**Figure 9.** First-order fitting. contour plots summarise expected overall (ECU + ITU) patient waiting times (in hours) with changing values of ECU beds ($B_{ECU}$) and arrival rate ($\lambda$), where the 81 data points produced with the FFD where fitted using a first-order meta-model. Asterisks represent the original decision determined from the what-if analysis.</figcaption></figure>
<figure><img src="image_40.png" /></figure>

* $\alpha \in [0, 0.4]$;

* $B_{ECU} \in \{9, 10, 11, 12, 13, 14, 15, 16, 17\}$

* $\lambda \in [2.4, 8.4]$;

* $\beta \in [0.1, 0.4]$.

Results are summarised in Figure 10. Results for an LHS design are very similar to those obtained with an FFD design; therefore, for this specific analysis, the more complex LHS design would not give us more information. Note that things might change if we considered a more sophisticated fitting, for instance based on artificial neural networks. A visual comparison with Figure 9 for the baseline arrival rate and with the plot on the right-hand side of Figure 8 suggests that a second-order meta-model might capture more accurately the relationship between the variables of interest as opposed to a first-order meta-model. Though more analyses should be run to test robustness of predictions for varying levels of arrival rate, we have more confidence in exploring scenarios represented in this meta-model as opposed to the first-order approximation. In terms of robustness of the decision about capacity, with high values of $\alpha$ (0.3), overall waiting time would likely benefit from an increase of beds in ECU (from 16 to 17) according to

the second-order meta-model as opposed to the what-if analysis. This capacity level would be best with the current or a higher level of arrival rates, whereas the chosen number of ECU beds could still be best in presence of low arrival rates. For lower values of $\alpha$, the current decision seems to be more robust to changes in the arrival rates.

## 6. Discussion

DoE methods are useful for understanding the impact of factors on a model's response, identifying optimal system configurations, and robust decision-making. In our literature search, we found 86 papers that used DES with DoE methods.

Despite calls for wider use of advanced DoE and metamodelling approaches that can take large numbers of factors into consideration, most articles only study small numbers of factors, using traditional DoE methods designed for physical experiments. Half of the articles included five factors or less, and full and fractional factorial designs were the most frequently used designs. These proportions did not seem to evolve with time.

### a) FFD – 2^nd^ order

$$ \alpha = 0 $$

<figure><img src="image_41.png" /></figure>

$$ \alpha = 0.15 $$

<figure><img src="image_42.png" /></figure>

$$ \alpha = 0.3 $$

<figure><img src="image_43.png" /></figure>

### b) LHS – 2^nd^ order

$$ \alpha = 0 $$

<figure><img src="image_44.png" /></figure>

$$ \alpha = 0.15 $$

<figure><img src="image_45.png" /></figure>

$$ \alpha = 0.3 $$

<figure><img src="image_46.png" /></figure>

**Figure 10.** Second-order fitting. contour plots summarise expected overall (ECU + ITU) patient waiting times (in hours) with changing values of ECU beds ($B_{ECU}$) and arrival rate ($\lambda$), where the 81 data points produced with the FFD (a) or the LHS (b) were fitted using a second-order meta-model. Asterisks represent the original decision determined from the what-if analysis.
<figure><img src="image_47.png" /></figure>

**6.1. Experimental design choices**

We found that modellers often chose full factorial designs for models with fewer than nine factors, because it allows the modeller to analyse all possible interactions between factors. However, as the number of factors increases, particularly beyond 35, orthogonal arrays become useful for efficiently minimising the number of experiments (Ilgin & Gupta, 2010, 2011a, 2011b; Tozanlı et al., 2020). In models with a large number of factors, e.g., ranging from 200 to 500, controlled sequential factorial designs are particularly effective (Shen & Wan, 2009). These designs are useful for screening important factors, eliminating noise factors, and ensuring a more focused approach to experimentation. The Taguchi methodology was chosen by nine papers for its ability to minimise the number of experimental runs (Wild & Pignatiello, 1991; Gamacheing et al., 2016; Low et al., 2018; Vargas & Roque, 2018; Arapoğlu, 2020; Keshtkar et al., 2020; Tozanlı et al., 2020; Mourtzis et al., 2021; Akl et al., 2024). The NOLH design was chosen for the same reason by modellers but was typically used when the model had approximately 10 or more factors. We assume that the factors analysed in the designs did not fully encompass all potential model variables, and additional factors could have been included. As computational constraints lessen with advances in computing power, future researchers may consider broadening the range of factors in their models.

Papers that did not create any response surfaces or metamodels typically used ANOVA, Tukey's tests, or Pareto plots to conduct a sensitivity analysis of their model's factors. When first- or second-order metamodels were created, ANOVA was usually used to analyse the statistical significance of factor interactions or the main effects of the factors.

**6.2. Sensitivity analysis techniques**

ANOVA is a straightforward method to identify which combinations of factors affect model output, but relies on assumptions that might not necessarily fit the model and must be verified, e.g., independence of observations, normally distributed residuals, and equality of variances (Corsini et al., 2022). ANOVA can be used after producing a set of simulation results based on some experimental design in order to identify which of the considered factors and factor combinations are statistically significant (Corsini et al., 2022). This is done by executing statistical hypothesis testing on the ratio “within-group variability”, to “within-group variability”, where different grouping criteria can be defined depending on the combinations of factors of interest.

Tukey's range test can be used on its own or following ANOVA to do pairwise comparisons between the

effects of different factors. For instance, after a given combination of multiple factors is identified as statistically significant through ANOVA, Tukey's range test can be applied on those factors to determine pairwise differences in their effects on model output (Baril et al., 2014). Tukey's range test, however, relies on the same assumptions as ANOVA, which may not be appropriate and should be verified prior to its application in order to avoid misleading results.

Pareto charts provide an additional criterion for selecting combinations of factors. Standardised effects, such as F-statistics, of all the tested combinations are reported in descending order, providing the decision-maker with a ranking of combinations among which the most relevant ones can be selected (e.g., Baesler et al., 2004; Salama & Eltawil, 2018).

**6.3. Metamodels and response surfaces**

Metamodelling and RSM provide “optimal solutions at a limited computational time” (Salama & Eltawil, 2018). In comparison to running simulations for every scenario, metamodels are advantageous in that they only need a small data set to create a model of the entire system. Metamodels are useful in speeding up the search of the solution space (Hoad et al., 2015) and give quick and meaningful reviews of the relationship between the responses and inputs. Response surfaces provide visual descriptions of these relationships. Metamodels can also be used to create user-friendly tools that allow users to change the configuration of the system and predict the response without the need to have a DES model run each time (e.g, Corsini et al., 2022). Despite the ease with which these tools facilitate solution space exploration, they are most valuable when accompanied by expert interpretation and analysis.

Choosing a meta-model requires the modeller to make certain assumptions about the system. For example, Kriging models are useful when the response surface can be modelled by a smooth and continuous function (Smew et al., 2013) and assume that the outputs are spatially correlated, i.e., response values closer together in the input space are more alike than those farther apart (Ankenman et al., 2010; Vicario et al., 2016). However, it is important to remember that metamodels are simply approximations of the system and require that the modeller understand how important high-order terms are in their system. In our case study, we could observe how choosing an appropriate model complexity is crucial for taking advantage of a meta-modelling approach. In cases where the complexity of the system cannot be easily captured by a meta-model approximation, it would be more convenient to use a less exhaustive but more reliable what-if analysis on selected scenarios of interest. Uncertainty quantification might also be helpful in
<figure><img src="image_48.png" /></figure>

deciding the right level of sophistication of the analysis. Techniques such as Bayesian regression or Bayesian neural networks have not been used in DoE applications across the literature analysed in this study. Considering them could help finding the right balance between number of experiments and accuracy of meta-model predictions.

Metamodels can be used to find the optimal combinations of parameters based on some criteria of interest. Simulation-optimisation enables a rather homogeneous exploration of the feasible set when this is defined either by exhaustive simulation planning or by the identification of a metamodel. Some papers used heuristic techniques to build a set of feasible solutions (e.g., Nasiri et al., 2017). However, by using this approach, the model might suffer from a non-exhaustive exploration of the parameter space.

## 6.4. Strengths and limitations

Our review method was transparent and replicable. It covered only one database, so we may have missed studies that combined DoE with DES. Nonetheless, all major simulation journals are indexed on WoS, as well as the Winter Simulation Conference proceedings where much DES research is shared, so we would not expect a major shift in our results if other databases were searched. Finally, by design our review did not include any papers that employ data farming techniques. This is another area which utilises DoE with DES models and has been reviewed in detail by Lechler et al. (2021) and Sanchez (2020).

## 6.5. Interpretation

The number of publications that include DES has been increasing steadily in recent years (Collins et al., 2023). In contrast, the number of studies that apply DoE to DES models remains limited.

Different factors may explain this situation. First, DES models are often developed in proprietary software that do not offer modules for DoE and cannot be easily chained with DoE packages developed in other systems (e.g., R or Python). This situation is likely to evolve as open-source software is increasingly used to develop DES (e.g Harper et al., 2023). It then becomes easier to combine DES models and DoE packages.

Second, DES practitioners may not be aware of more advanced DoE techniques. In Hoad et al.'s survey, only half of respondents had any practical experience of DoE, which may not go further than full factorial designs (Hoad et al., 2015). DoE remains poorly addressed in DES textbooks, and even in simulation conferences, tutorials still dedicate a significant amount of their content to simple designs (e.g., Law, 2017; Sanchez et al., 2020). DoE seems to remain

a specialist skill among DES modellers. Yet, our case study shows that this does not need to be the case: a range of packages exist in common coding languages (we used R in our case study, but Python also offers several options). Although some statistical understanding is required, the implementation itself becomes much easier. Yet, again, modellers who use commercial DES software must find a way to interface them with a programming language.

Some simulation projects may not require advanced DoE techniques. Sometimes, we only want to know about one or a few scenarios. In this situation, modellers may be reluctant to bother with advanced experimentation. Yet, even then, DoE can help modellers learn about their model and assess the robustness of their findings (J. Kleijnen et al., 2005). We demonstrated this in the case study, in which we used Latin Hypercube Sampling and full factorial designs to augment the DES sensitivity analysis. In Hoad et al.'s survey, the time spent on experimentation was only marginally longer when modellers used DoE. There does not seem to be a downside to using DoE, whereas the benefits are clear.

In this situation, how can we make progress? Nine years later, we can only reiterate Hoad et al.'s recommendation that simulation educators include DoE in their teaching (Hoad et al., 2015). Simulation courses could also be paired with quality engineering courses, where DoE is likely to be discussed. But in the latter option, the discussion is likely to focus on DoE for physical experiments, whereas simulation can benefit from specific DoE approaches (J. Kleijnen et al., 2005). Journal editors and reviewers also play a role. Systematic strategies for experimenting with models could be requested, or at least encouraged, when submitting DES case studies. The STRESS guidelines for reporting the results of empirical simulation studies (Monks et al., 2019) only specify that experiments should be fully reported and do not include recommendations on how they should be designed.

## 7. Conclusion

With computer simulations, it is easy to run experiments with many factors. Simulation packages are more readily available with open-sourced software, such as Python or R. These make it easy to incorporate complicated experimental designs into projects. Without the constraints traditionally associated with physical manufacturing facilities, modellers are able to conduct more in-depth analyses. However, we found that this is not common among researchers.
<figure><img src="image_49.png" /></figure>

# Acknowledgements

This project was funded through a PhD Teaching Assistantship by the Department of Mathematics at UCL.

# Disclosure statement

No potential conflict of interest was reported by the author(s).

# ORCID

Amalia Gjerloev http://orcid.org/0000-0002-2914-7992

Luca Grieco http://orcid.org/0000-0002-8733-6062

Guillaume Lamé http://orcid.org/0000-0001-9514-1890

Sonya Crowe http://orcid.org/0000-0003-1882-5476

# References

Akl, A. M., Rahman, H. F., Chakraborty, R. K., & El Sawah, S. (2024). An assets maintenance-workforce planning problem under uncertainty: A chance constraints assisted simulation-optimization approach. *Simulation Modelling Practice and Theory*, 130, 102839.

Altiok, T., & Melamed, B. (2007). *Simulation modeling and analysis with Arena*. Aca-demic Press.

Amaral, J. V., Montevechi, S. D., Miranda, R. D. C., & Junior, W. T. D. S. (2022). Metamodel-based simulation optimization: A systematic literature review. *Simulation Modelling Practice and Theory*, 114, 102403. https://doi.org/10.1016/j.simpat.2021.102403

Amaral, J. V. S. D., Miranda, R. D. C., Montevechi, J. A. B., Santos, C. H. D., & Brito, F. D. O. (2025). Adaptive metamodeling-based simulation optimisation. *Journal of the Operational Research Society*, 76(6), 1156-1176. https://doi.org/10.1080/01605682.2024.2415474

Ankenman, B., Nelson, B. L., & Staum, J. (2010). Stochastic kriging for simulation metamodeling. *Operations Research*, 58(2), 371-382. https://doi.org/10.1287/opre.1090.0754

Arapoğlu, R. (2020, October 26 - 28). Productivity improvement at motor vehicle inspection station by using taguchi approach. 5th EurAsia Waste Management Symposium (EWMS), Istanbul, Turkey, 441-445.

Assid, M., Gharbi, A., & Hajji, A. (2015). Production planning and opportunistic preventive maintenance for unreliable one-machine two-products manufacturing systems. *IFAC-Papersonline*, 48(3), 478-483. https://doi.org/10.1016/j.ifacol.2015.06.127

Baesler, F. F., Araya, E., Ramis, F. J., & Sepúlveda, J. A. (2004). The use of simulation and design of experiments for productivity improvement in the sawmill industry. *Proceedings of the 2004 Winter Simulation Conference*, 2004. (Vol. 2. pp. 1218-1221). IEEE.

Baril, C., Gascon, V., & Cartier, S. (2014). Design and analysis of an outpatient orthopaedic clinic performance with discrete event simulation and design of experiments. *Computers & Industrial Engineering*, 78, 285-298. https://doi.org/10.1016/j.cie.2014.05.006

Barton, R. R. (2009). Simulation optimization using metamodels. *Proceedings of the 2009 winter simulation conference (WSC)* (pp. 230-238). IEEE.

Box, G. E., & Behnken, D. W. (1960). Some new three level designs for the study of quantitative variables.

Technometrics, 2(4), 455-475. https://doi.org/10.1080/00401706.1960.10489912

Box, G. E., & Wilson, K. B. (1951). On the experimental attainment of optimum conditions. *Journal of the Royal Statistical Society: Series B (Methodological)*, 13(1), 1-38. https://doi.org/10.1111/j.2517-6161.1951.tb00067.x

Carson, J. S., Nelson, B. L., & Nicol, D. M. (2005). Discrete-event system simulation. In Banks, J. (Ed.), *Prentice-hall international series in industrial and systems engineering* (4th ed., pp. 271). Pearson Prentice Hall.

Červeňanská, Z., Kotianová, J., Važan, P., Juhásová, B., & Juhás, M. (2020). Multi-objective optimization of production objectives based on surrogate model. *Applied Sciences*, 10(21), 7870. https://doi.org/10.3390/ap10217870

Chackelson, C., Errasti, A., Ciprés, D., & Lahoz, F. (2013). Evaluating order picking performance trade-offs by configuring main operating strategies in a retail distributor: A design of experiments approach. *International Journal of Production Research*, 51(20), 6097-6109. https://doi.org/10.1080/00207543.2013.796421

Chan, K., & Spedding, T. (2001). On-line optimization of quality in a manufacturing system. *International Journal of Production Research*, 39(6), 1127-1145. https://doi.org/10.1080/00207540010023015

Cherkassky, A. (2011). Neural network meta-model of fibrous materials based on discrete-event simulation. part 2: Neural network meta-model. *The Journal of the Textile Institute*, 102(6), 475-482. https://doi.org/10.1080/00405000.2010.488079

Collins, A. J., Sabz Ali Pour, F., & Jordan, C. A. (2023). Past challenges and the future of discrete event simulation. *The Journal of Defense Modeling and Simulation*, 20(3), 351-369. https://doi.org/10.1177/15485129211067175

Corsini, R., Costa, A., Fichera, S., Pluchino, A., & Parrinello, V. (2022). System design of outpatient chemotherapy oncology departments through simulation and design of experiments. *International Journal of Management Science and Engineering Management*, 19 (1), 1-14. https://doi.org/10.1080/17509653.2022.2134223

Fonseca, D. J., Navaresse, D. O., & Moynihan, G. P. (2003). Simulation metamodeling through artificial neural networks. *Engineering Applications of Artificial Intelligence*, 16(3), 177-183. https://doi.org/10.1016/S0952-1976(03)00043-5

Fu, J., Zhang, J., Ding, G., Qin, S., & Jiang, H. (2021). Determination of vehicle requirements of agv system based on discrete event simulation and response surface methodology. *Proceedings of the Institution of Mechanical Engineers, Part B: Journal of Engineering Manufacture*, 235(9), 1425-1436. https://doi.org/10.1177/0954405421995602

Gamacheing, S., & Abdulnouring, G. (2016). Dynamic manufacturing cells and SME network: key success factors. *IFAC-PapersOnLine*, 49(12), 851-856.

Greasley, A., & Edwards, J. S. (2021). Enhancing discrete-event simulation with big data analytics: A review. *Journal of the Operational Research Society*, 72 (2), 247-267. https://doi.org/10.1080/01605682.2019.1678406

Grieco, L. (2024). *Doe analysis ecu library*. https://github.com/lucagrieco19/DoEanalysisECU

Harper, A., Pitt, M., & Monks, T. (2023). Open-source modeling for orthopedic elective capacity planning using discrete-event simulation. *2023 Winter Simulation Conference (WSC)*, (pp. 1077-1087). IEEE.
<figure><img src="image_50.png" /></figure>

Hill, R. R., Gutman, A. J., Moulder, R. D., Stafford, T. D., & Bush, K. R. (2015). Case studies: Definitive screening applied to a simulation study of the f100-229 engine repair network. *Quality Engineering, 27*(4), 424-436. https://doi.org/10.1080/08982112.2015.1023315

Hoad, K., Monks, T., & O'Brien, F. (2015). The use of search experimentation in discrete-event simulation practice. *Journal of the Operational Research Society, 66*(7), 1155-1168. https://doi.org/10.1057/jors.2014.79

Hollocks, B. W. (2001). Discrete-event simulation: An inquiry into user practice. *Simulation Practice and Theory, 8*(6), 451-471. https://doi.org/10.1016/S0928-4869(01)00028-3

Hussein, M., Karam, A., Eltoukhy, A. E., Darko, A., & Zayed, T. (2023). Optimized multimodal logistics planning of modular integrated construction using hybrid multi-agent and metamodeling. *Automation in Construction, 145*, 104637. https://doi.org/10.1016/j.aut con.2022.104637

Ilgin, M. A., & Gupta, S. M. (2010). Use of sensor embedded products for end of life processing. *Proceedings of the 2010 Winter Simulation Conference* (pp. 1584-1591). IEEE.

Ilgin, M. A., & Gupta, S. M. (2011a). Evaluating the impact of sensor-embedded products on the performance of an air conditioner disassembly line. *The International Journal of Advanced Manufacturing Technology, 53* (9-12), 1199-1216. https://doi.org/10.1007/s00170-010-2891-0

Ilgin, M. A., & Gupta, S. M. (2011b). Recovery of sensor embedded washing machines using a multi-kanban controlled disassembly line. *Robotics and computer-Integrated Manufacturing, 27*(2), 318-334. https://doi.org/10.1016/j.rcim.2010.07.016

Jia, Y., Tian, H., Chen, C., & Wang, L. (2017). Predicting the availability of production lines by combining simulation and surrogate model. *Advances in Production Engineering & Management, 12*(3), 285. https://doi.org/10.14743/apem2017.3.259

Jones, B., & Nachtsheim, C. J. (2011). A class of three-level designs for definitive screening in the presence of second-order effects. *Journal of Quality Technology, 43* (1), 1-15. https://doi.org/10.1080/00224065.2011.11917841

Kang, K., & McDonald, M. (2010). Impact of logistics on readiness and life cycle cost: A design of experiments approach. *Proceedings of the 2010 winter simulation conference* (pp. 1336-1346). IEEE.

Keshtkar, L., Rashwan, W., Abo-Hamad, W., and Arisha, A. (2020). A hybrid system dynamics, discrete event simulation and data envelopment analysis to investigate boarding patients in acute hospitals. *Operations Research for Health Care, 26*, 100266.

Kleijnen, J. P. (2009). Kriging metamodeling in simulation: A review. *European Journal of Operational Research, 192* (3), 707-716. https://doi.org/10.1016/j.ejor.2007.10.013

Kleijnen, J. P. (2015). Design and analysis of simulation experiments. International workshop on simulation. (pp. 3-22). Springer International Publishing.

Kleijnen, J. P. C. (2005). An overview of the design and analysis of simulation experiments for sensitivity analysis. *European Journal of Operational Research, 164* (2), 287-300. https://doi.org/10.1016/j.ejor.2004.02.005

Kleijnen, J. P. C. (2008a). Simulation experiments in practice: Statistical design and regression analysis. *Journal of Simulation, 2*(1), 19-27. https://doi.org/10.1057/palgrave.jos.4250032

Kleijnen, J. P. C. (2008b). Simulation experiments in practice: Statistical design and regression analysis. *Journal of Simulation, 2*(1), 19-27. https://doi.org/10.1057/palgrave.jos.4250032

Kleijnen, J. P. C. (2009). Kriging metamodeling in simulation: A review. *European Journal of Operational Research, 192*(3), 707-716. https://doi.org/10.1016/j.ejor.2007.10.013

Kleijnen, J. P. C. (2017). Regression and kriging metamodels with their experimental designs in simulation: A review. *European Journal of Operational Research, 256*(1), 1-16. https://doi.org/10.1016/j.ejor.2016.06.041

Kleijnen, JPC, & RG, Sargent. (2000). A methodology for fitting and validating metamodels in simulationtwo anonymous referees' comments on the first draft lead to an improved organization of our paper.1. *European Journal of Operational Research, 120*(1), 14-29. https://doi.org/10.1016/S0377-2217(98)00392-0

Kleijnen, J., Sanchez, S. M., Lucas, T. W., & Cioppa, T. M. (2005). A user's guide to the brave new world of designing simulation experiments: State-of-the-art review. *INFORMS Journal on Computing, 17*(3), 263-289. https://doi.org/10.1287/ijoc.1050.0136

Law, A. M. (2013). *Simulation modeling and analysis. McGraw-Hill series in industrial engineering and management science* (fifth ed.). McGraw-Hill Education.

Law, A. M. (2017). A tutorial on design of experiments for simulation modeling. *2017 Winter Simulation Conference (WSC), IEEE*. (pp. 550-564).

Lechler, T., Sjarov, M., & Franke, J. (2021). Data farming in production systems-a review on potentials, challenges and exemplary applications. *Procedia CIRP, 96*, 230-235. https://doi.org/10.1016/j.procir.2021.01.156

Marlin, B., & Sohn, H. (2016). Using data in conjunction with designs of experiments: An approach to assess simulated futures in the Afghan educational system. *Journal of Simulation, 10*(4), 272-282. https://doi.org/10.1057/jos.2015.14

Monks, T., Currie, C. S. M., Onggo, B. S., Robinson, S., Kunc, M., & Taylor, S. J. E. (2019). Strengthening the reporting of empirical simulation studies: Introducing the stress guidelines. *Journal of Simulation, 13*(1), 55-67. https://doi.org/10.1080/17477778.2018.1442155

Montevechi, J. A. B., de Almeida Filho, R. G., Paiva, A., Costa, R., & Medeiros, A. L. (2010). Sensitivity analysis in discrete-event simulation using fractional factorial designs. *Journal of Simulation, 4*(2), 128-142. https://doi.org/10.1057/jos.2009.23

Montgomery, D. C. (2017). *Design and analysis of experiments*. John wiley & sons.

Mourtzis, D., Angelopoulos, J., & Panopoulos, N. (2021). Robust engineering for the design of resilient manufacturing systems. *Applied Sciences, 11*(7), 3067. https://doi.org/10.3390/app11073067

Nasiri, M. M., Yazdanparast, R., & Jolai, F. (2017). A simulation optimisation approach for real-time scheduling in an open shop environment using a composite dispatching rule. *International Journal of Computer Integrated Manufacturing, 30*(12), 1239-1252. https://doi.org/10.1080/0951192X.2017.1307452

Özdemirel, N. E., Yurttas, G. Y., & Köksal, G. (1996). Computer-aided planning and design of manufacturing simulation experiments. *Simulation, 67*(3), 171-191. https://doi.org/10.1177/003754979606700304

Robinson, S. (2004). *Simulation: The practice of model development and use*. John Wiley & Sons.

Rossetti, M. D. (2016). *Simulation modeling and Arena* (second ed.). John Wiley & Sons, Inc.
<figure><img src="image_51.png" /></figure>

Salama, S., & Eltawil, A. B. (2018). A decision support system architecture based on simulation optimization for cyber-physical systems. *Procedia Manufacturing*, 26, 1147-1158. https://doi.org/10.1016/j.promfg.2018.07.151

Sanchez, S. M. (2018). Data farming: Better data, not just big data. *2018 Winter Simulation Conference (WSC)* (pp. 425-439). IEEE.

Sanchez, S. M. (2020). Data farming: Methods for the present, opportunities for the future. *The ACM Transactions on Modeling and Computer Simulation (TOMACS)*, 30 (4), 1-30. https://doi.org/10.1145/3425398

Sanchez, S. M., Lucas, T. W., Sanchez, P. J., Nannini, C. J., Wan, H., & Hinkelmann, K. (2012). *Design and Analysis of Experiments*, 3, 413-441. https://doi.org/10.1002/9781118147634

Sanchez, S. M., & Sánchez, P. J. (2017). Better big data via data farming experiments. *Advances in Modeling and Simulation: Seminal Research from 50 Years of Winter Simulation Conferences* (pp. 159-179). Springer.

Sanchez, S. M., Sanchez, P. J., & Wan, H. (2020). Work smarter, not harder: A tutorial on designing and conducting simulation experiments. *Proceedings of the Winter Simulation Conference* (pp. 1128-1142). IEEE Press.

Shen, H., & Wan, H. (2009). Controlled sequential factorial design for simulation factor screening. *European Journal of Operational Research*, 198(2), 511-519. https://doi.org/10.1016/j.ejor.2008.09.005

Smew, W., Young, P., & Geraghty, J. (2013). Supply chain analysis using simulation, gaussian process modelling and optimisation. *International Journal of Simulation*

Modelling, 12(3), 178-189. https://doi.org/10.2507/IJSIMM12(3)4.239

Tozanlı, Ö., Kongar, E., & Gupta, S. M. (2020). Evaluation of waste electronic product trade-in strategies in predictive twin disassembly systems in the era of blockchain. *Sustainability*, 12(13), 5416. https://doi.org/10.3390/su12135416

Vargas, J. and Calvo, R. (2018). Joint optimization of process flow and scheduling in service- oriented manufacturing systems. *Materials*, 11(9):1559.

Vicario, G., Craparotta, G., & Pistone, G. (2016). Meta-models in computer experiments: Kriging versus artificial neural networks. *Quality and Reliability Engineering International*, 32(6), 2055-2065. https://doi.org/10.1002/qre.2026

Wan, H., & Ankenman, B. E. (2007). Two-stage controlled fractional factorial screening for simulation experiments. *Journal of Quality Technology*, 39(2), 126-139. https://doi.org/10.1080/00224065.2007.11917680

Wild, R. H., & Pignatiello, J. J., Jr. (1991). An experimental design strategy for designing robust systems using discrete-event simulation. *Simulation*, 57(6), 358-368. https://doi.org/10.1177/003754979105700603

Yang, F. (2010). Neural network metamodeling for cycle time-throughput profiles in manufacturing. *European Journal of Operational Research*, 205(1), 172-185. https://doi.org/10.1016/j.ejor.2009.12.026

Zhang, Y., & Brown, D. (2014). Simulation optimization of police patrol districting plans using response surfaces. *Simulation*, 90(6), 687-705. https://doi.org/10.1177/0037549714533159