# Practical Considerations about Error Analysis for Discrete Event Simulations Model

Giovanni Davoli, Peter Nielsen, Gabriele Pattarozzi, Riccardo Melloni

► To cite this version:

Giovanni Davoli, Peter Nielsen, Gabriele Pattarozzi, Riccardo Melloni. Practical Considerations about Error Analysis for Discrete Event Simulations Model. 19th Advances in Production Management Systems (APMS), Sep 2012, Rhodes, Greece. pp.707-713, 10.1007/978-3-642-40361-3_90. hal-01470689

HAL Id: hal-01470689

https://inria.hal.science/hal-01470689

Submitted on 17 Feb 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Practical considerations about error analysis for discrete event simulations model

Giovanni Davoli¹, Peter Nielsen², Gabriele Pattarozzi¹, Riccardo Melloni¹.

¹ Department of Mechanical and Civil Engineering (DIMeC), University of Modena and Reggio Emilia; via Vignolese 905, 41100, Modena, Italy

² Department of Production, Aalborg University, Fibigerstrede 16, DK 9220 Aalborg, Denmark

giovanni.davoli@unimore.it

**Abstract.** The purpose of making efficient and flexible manufacturing systems is often related to the possibility to analyze the system considering at the same time a wide number of parameters and their interactions. Simulation models are proved to be useful to support and drive company management in improving the performances of production and logistic systems. However, to achieve the expected results, a detailed model of the production and logistic system is needed as well as a structured error analysis to guarantee results reliability. The aim of this paper is to give some practical guide lines in order to drive the error analysis for discrete event stochastic simulation model that are widely used to study production and logistic system.

**Keywords:** discrete event, simulation model, error analysis, stochastic model.

## 1 Introduction

Stochastic, discrete events, simulation models are widely used to study production and logistic system. Apart from the development, one of the main problem of this approach is to perform the error analysis on the outputs of the simulation model. Simulation experiments are classified as either terminating or non-terminating as far as the goal of the simulation is concerned (Law and Kelton, 2000), (Fishman, 2000). If we limit our interests on non-terminating simulation, the error analysis can be split into two different parts. The first part consists of individuating the initial transient period and the confidence interval of the outputs. The second part consists of estimating how the transient period and the outputs confidence interval varies when the initial model scenario is changed. The first part of the problem is widely studied, Kelton (1983-1989), Schruben (1982-1983), Welch (1982), Vassilacopoulos (1989) White (1997), and many methods are provided to determinate the transient period often related to output stability, that can be quantified in different ways. Between the proposed techniques Mean Squared Pure Error method, Mosca et al. (1985-1992), should be reminded as a practical method useful to determinate both transient period and confidence interval. On the other hand the second part of error analysis problem is not commonly addressed directly as reported in the recent work of Sandikc (2006)
that tries to fill the gap for the initial transient period for simulation model addressing production lines. The variance of outputs confidence interval between different scenario is often faced with the hypothesis that it is normally distributed around a central value used in the reference scenario according with the basic theory of statistics (Box et al. 2013). But in many practical cases there is no evidences that this hypothesis is correct and, moreover, the significance of central value, for the reference scenario, is lost. In fact in some recent simulation handbook (Chung 2004) the advice to quantify the confidence interval for all different simulated scenario is given.

## 2 Purpose

The aim of this paper is to give some practical guidelines in order to drive the error analysis for discrete event stochastic simulation models. The paper is focused on the study of confidence interval variance related to the variance of simulated scenario. Nowadays, in many practical applications, the calculation potential is large enough to perform “long” simulation run in order to assure to exceed the initial transient period. Much more important is to determinate the confidence interval for the outputs in different simulated scenario, because overestimate or underestimate these confidence intervals can drive analysts towards a wrong interpretation of the results.

## 3 Methodology

To address the aim of the paper a quite simple discrete event simulation model is considered and the MSPE (1) is used to estimate outputs confidence interval. Then the simulation are performed according to different scenario and the variance of confidence interval is studied for different outputs.

$$MSPE_i = \frac{\sum_{j=1}^{r}(y_{ji}-Y_i)^2}{r-1} \quad (1)$$

This paper is grounded on a discrete events simulation model reproducing a re-order point logistic system, in particular a single-item fixed order quantity system also known as: Economic Order Quantity (EOQ) model. The economic order quantity (EOQ), first introduced by Harris (1913), and developed by Brown (1963) and Bather (1966) with stochastic demand, is a well-known and commonly used inventory control techniques reported in a great variety of hand book, for example: Tersine (1988) and Ghiani (2004). The notation used in this paper is illustrated in table 1.

<footnote>* i: day; y<sub>ji</sub>: output value at day i, replication j; Y<sub>i</sub>: output mean at day i, on r replications.</footnote>
<table><caption>**Table 1.** Symbol and definitions</caption><thead><tr><th>Symbol</th><th>Unit</th><th>Definition</th></tr></thead><tbody><tr><td>N</td><td>Day</td><td>Number of days for simulation</td></tr><tr><td>D<sub>i</sub></td><td>Unit/day</td><td>Mean demand per day in units</td></tr><tr><td>L<sub>t</sub></td><td>Day</td><td>Mean lead time in day</td></tr><tr><td>C<sub>o</sub></td><td>Euro/order</td><td>Single order cost in euro</td></tr><tr><td>C<sub>s</sub></td><td>Euro/unit*year</td><td>Stock cost in euro per unit per year</td></tr><tr><td>S<sub>s</sub></td><td>Unit</td><td>Safety stocks in unit</td></tr></tbody></table>

## 3.1 Simulation model

The simulation model was developed according with the standard EOQ model for single item. A set of stochastic functions, developed in SciLab environment, are used to generate the demand that activates the model. The simulation model was tested performing standard EOQ model with normal distributed demand (where $\sigma_d$ is demand standard deviation) and normal distributed lead time (where $\sigma_t$ is lead time standard deviation). The parameters set used in the reference scenario are illustrated in table 2.

<table><caption>**Table 2.** Used parameters set</caption><thead><tr><th>Parameter</th><th>Set value</th></tr></thead><tbody><tr><td>D<sub>i</sub></td><td>1.000,00</td></tr><tr><td>σ<sub>d</sub></td><td>300,00</td></tr><tr><td>L<sub>t</sub></td><td>7,00</td></tr><tr><td>σ<sub>t</sub></td><td>2,00</td></tr><tr><td>C<sub>o</sub></td><td>1.000,00</td></tr><tr><td>C<sub>s</sub></td><td>1,00</td></tr><tr><td>Imposed SL</td><td>0,95</td></tr></tbody></table>

To evaluate model performances, in terms of achieved service level, a set of 4 Key Performance Indicators (KPI) is defined. The used KPI are illustrated in table 3.

<table><caption>**Table 3.** Used KPI</caption><thead><tr><th>KPI</th><th>Unit</th><th>Definition</th></tr></thead><tbody><tr><td>SL1</td><td>%</td><td>1-Number of stock-out in days per day</td></tr><tr><td>SL2</td><td>%</td><td>1-Number of stock-out in units per day</td></tr><tr><td>SL3</td><td>%</td><td>1-Number of stock-out in units per day during lead time</td></tr><tr><td>SL4</td><td>%</td><td>1-Number of stock-out event during lead time period</td></tr></tbody></table>
## 3.2 Design of the experiments

To investigate the influence of different parameters on confidence intervals four factors are considered. These four factors are:

* Demand distribution;

* Lead time distribution;

* Ratio $C_o/C_s$;

* SS, safety stocks.

A full factorial experiment with three levels is used in this paper. Four factors and three levels give $3^4 = 81$ combinations. For each combination a number of 5 replications were conducted for a number of 405 simulations. The three settings for the four factors are shown in table 4.

<table><caption>**Table 4.** Factors setting</caption><thead><tr><th></th><th>Low (-1)</th><th>Mean (0)</th><th>High (+1)</th></tr></thead><tbody><tr><th>D</th><td>Normal distribution, mean = 1,000 units/day, standard deviation = 300 units/day</td><td>Uniform distribution, minimum = 500 units/day, maximum = 1,500 units/day</td><td>Exponential distribution, mean = 1,000 units/day</td></tr><tr><th>Lt</th><td>Normal distribution, mean = 7 day, standard deviation = 2 day</td><td>Uniform distribution, minimum = 1 day, maximum = 13 day</td><td>Exponential distribution, mean = 7 day</td></tr><tr><th>C<sub>o</sub>/C<sub>s</sub></th><td>100</td><td>1.000</td><td>1.900</td></tr><tr><th>SS</th><td>0 units</td><td>1.000 units</td><td>2.000 units</td></tr></tbody></table>

## 4 Findings

The presented experiments are evaluated in terms of stability of the results and confidence interval width for all considered KPI. The simulations are conducted for a length of 1.000 days and this guarantee the stability of outputs for all KPI. Initial transient period length varies according with different parameters set and the variance is more significant for certain KPI, as shown in figure 1.
<figure><img src="image_2.png" /><figcaption>Figure.1 MSPE for KPI SL1 and SL3</figcaption></figure>

To evaluate the significance of confidence intervals the results are presented for each KPI as the ratio between half interval and the mean for each KPI. Confidence half intervals are calculated for a 95% level of significance according with (2).

$$I_{\frac{1}{2}} = Z_{\alpha} \frac{\sigma_{KPIn}}{\sqrt{r}} \quad (2)$$

* Zα/2 normalized standard variable, σKPIn standard deviation for KPIn and r is replication number.

<table><caption>Table 5. ANOVA test P-value results, codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05</caption><thead><tr><th>Factors</th><th>LS1</th><th>LS2</th><th>LS3</th><th>LS4</th></tr></thead><tbody><tr><td>Demand distribution</td><td>3,15E-14 ***</td><td>4,99E-14 ***</td><td>4,85E-15 ***</td><td>0,0002638 ***</td></tr><tr><td>Lead time distribution</td><td>&lt; 2.2e-16 ***</td><td>&lt; 2.2e-16 ***</td><td>&lt; 2.2e-16 ***</td><td>0,3792613 ***</td></tr><tr><td>Safety Stocks</td><td>0,004364 **</td><td>0,013121 *</td><td>0,0001933 ***</td><td>8,62E-09 ***</td></tr><tr><td>Ratio Co/Cs</td><td>0,132156 ***</td><td>0,126539 ***</td><td>&lt; 2.2e-16 ***</td><td>3,49E-07 ***</td></tr><tr><td>Demand dist.: Lead time dist.</td><td>&lt; 2.2e-16 ***</td><td>&lt; 2.2e-16 ***</td><td>4,03E-16 ***</td><td>0,1436708 ***</td></tr><tr><td>Demand dist.: Safety Stocks</td><td>0,639589 ***</td><td>0,952472 ***</td><td>0,8427638 ***</td><td>0,9926262 ***</td></tr><tr><td>Demand dist.: Ratio Co/Cs</td><td>0,016213 *</td><td>0,031195 *</td><td>0,0091386 **</td><td>0,0912483 ***</td></tr><tr><td>Lead time dist.: Safety Stocks</td><td>0,570304 ***</td><td>0,801933 ***</td><td>0,0424879 *</td><td>0,2119648 ***</td></tr><tr><td>Lead time dist.: Ratio Co/Cs</td><td>0,025063 *</td><td>0,009775 **</td><td>&lt; 2.2e-16 ***</td><td>0,1682513 ***</td></tr><tr><td>Safety Stocks: Ratio Co/Cs</td><td>0,92423 ***</td><td>0,918936 ***</td><td>0,9432251 ***</td><td>0,8817591 ***</td></tr></tbody></table>

The ANOVA test reveals that the considered factors have different impact on confidence interval. Demand and lead time distribution have a very strong effects in comparison with the other parameters and even their interaction is important, as shown in figure 2 for SL1.
<figure><img src="image_3.png" /><figcaption>**Figure.2** Confidence half interval (min – max), in terms of %, for SL1 and SL4</figcaption></figure>

**5 Conclusions**

The case study presented here can be used to make some practical considerations to support error analysis for discrete event simulation models. First, a “long” simulation period, in order to pass the initial transient period, is relatively easy to set, even if different behavior have been observed for different KPI. Second, the initial transient period and the related confidence interval depend in a very different way by the considered parameters. In particular, for numeric parameters, the hypothesis that confidence interval variance is normal distributed around a central value calculated in the reference scenario is almost verified. On the other hand, when the studied parameters are not numerical, for example distribution type as in the considered case study, the confidence interval must be re-calculated in each scenario because the variance could be high and the interaction are almost unpredictable. So, in practice, the effort to check the confidence interval related to discrete event simulation should be done when the modified parameters are not simply numeric. This kind of analysis, thanks to the actual computational resource, is not prohibitive in terms of time when we manage a rather simple model.

**7 Limitation and further work**

The number of replications for each scenario provided in the DOE is fixed, a deeper study about this aspect should be investigate.

**References**

1. Bather, J.A.: A continuous time inventory model. J. Appl. Prob. 3, pp: 538—549 (1966)

2. Box G. E. P., Hunter W. e Hunter J. S., Statistics for experimenters. John Wiley & Sons (1978)

3. Brown, R.G.: Smoothing, Forecasting and Prediction of Discrete Time Series. Englewood Cliffs, N.J., Prentice-Hall (1963)
4. Chung A. C. "Simulation Modeling Handbook – A Pratical Approach". Industrial and Manufacturing Engineering Series, Series Editor, CRC Press (2004)

5. Fishman, G.S. Discrete Event Simulation: Modeling, Programming, and Analysis. Springer-Verlag (2001)

6. Ghiani, G., Laporte, G., Musmanno, R.: Introduction to logistics systems planning and control, West Sussex England, John Wiley & Sons Ltd (2004)

7. Harris, F.W., 1913. How many parts to make at once. Factory, The Magazine of Management 10(2), 135–136, 152; reprinted in Operations Research 38 (6), November–December (1990)

8. Kelton, W.D. Random initialization methods in simulation. IIE Transactions 21 (4), 355–367. (1989)

9. Kelton, W.D., Law, A.M. A new approach for dealing with the startup problem in discrete event simulation. Naval Research Logistics Quarterly 30, 641–658 (1983)

10. Law, A.M., Kelton, W.D. Simulation Modeling and Analysis, third ed. McGraw-Hill (2000)

11. Mosca R., Giribone P. Teoria degli esperimenti e simulazione. Quaderni di gestione degli impianti industriali, Università di Genova (1985)

12. Mosca R., Giribone P., Schenone M. Integrated management of a bishuttle FMS using discrete/stochastic simulator. Computer-Integrated Manufacturing Systems, Vol. 5 No. 2 (1992)

13. Sandıkc B., Sabuncuoglu I. Analysis of the behavior of the transient period in non-terminating simulations. European Journal of Operational Research Vol. 173 (2006) pp. 252-267, 2006

14. Schruben, L.W. Detecting initialization bias in simulation output. Operations Research 30, 569–590 (1982)

15. Schruben, L.W., Singh, H., Tierney, L. Optimal test for initialization bias in simulation output. Operations Research 31, 1167–1178 (1983)

16. Tersine R.J.: Principles of inventory and materials management. North-Holland, Elsevier Science Publishing Co., Inc. (1988)

17. Vassilacopoulus, G. Testing for initialization bias in simulation output. Simulation 52, 151–153 (1989)

18. Welch, P.D. A graphical approach to the initial transient problem in steady-state simulations. In: Proceedings of the 10th IMACS World Congress on Systems, Simulation, and Scientific Computation, Montreal, pp. 219–221 (1982)

19. White Jr., K.P. An effective truncation heuristic for bias reduction in simulation output. Simulation 69 (6), 323–334 (1997)