# Integrated Robotic and Network Simulation Method
## 3 IRoNS Method
### Base Simulation--Validation  
After finishing the first simulation implementation, referred as a candidate Base Simulation, the simulation needs to pass a validation process. As there are diverse types of algorithms, it is not trivial validating them all at the same time, and it would probably require a testbed [33] or real robots to reproduce real-world results for validation, which may not be available.  
One approach is to consider individual domain validations and assume composability, i.e., the individual validations will remain valid upon integration. Thus, the resulting simulation should give some indications of the overall behavior of the system and possible algorithms interactions, even if the results are not precisely accurate.  
There are several techniques already developed for simulation validation [30; 33], but we adopt a combination of a simple four-step ABMS validation technique [21] with a statistical analysis via confidence intervals (CI) from VV&T research [22].  
The four steps in [21] consist of face validation, sensitivity analysis, calibration, and statistical validation. Face validation consists in a human expert analyzing the overall behavior of the system,making the necessary modifications to achieve expected results or to eliminate visibly bad results. Sensitivity analysis aims at assessing the impact that parameter changes have on the simulation behavior. The third step is the calibration, adjusting parameters to increase model accuracy and reevaluate its final values.  
The final step is the statistical validation, for which we use a confidence interval technique using the errors between multiple simulation instances [22]. For this purpose, we formalize the technique as follows: Let \(R_{1}\) and \(R_{2}\) be the statistical results from the same situation simulated on different simulation frameworks, Simulator 1 and Simulator 2 respectively, \(Z_{j}=R_{1j}-R_{2j}\) as the error achieved in an experiment \(j\). The mean \(\overline{Z}(n)\) and variance \(\hat{Var}\Big{[}\overline{Z}(n)\Big{]}\) are then computed for a given number \(n\) of experiments (samples).  
The confidence interval (\(CI_{z}\)) is defined by (1) using \(\overline{Z}(n)\) with \(\hat{Var}\Big{[}\overline{Z}(n)\Big{]}\) used to determine the half-length (\(L_{z}\)) of the interval (2) in terms of \(t_{n-1,1-\frac{\alpha}{2}}\) critical points of a \(t\)-_student_ distribution, obtained from a \(t\)-_student_ table [22] as function of \(n-1\) samples and an auxiliary constant \(\alpha\), determined by the desirable confidence C\({}_{d}\) (3).  
\[CI_{Z}=\Big{[}\overline{Z}(n)+L_{Z}\ \ \,\ \ \overline{Z}(n)-L_{Z}\Big{]} \tag{1}\]  
\[L_{Z}=t_{n-1,1-\frac{\alpha}{2}}\sqrt{\hat{Var}\Big{[}\overline{Z}(n)\Big{]}} \tag{2}\]  
\[\alpha=2\cdot(1-\frac{C_{d}}{100}) \tag{3}\]  
The resulting interval can be assessed in two ways: statistical and scale significances. If the confidence interval contains zero, it means that, with a determined confidence, the true mean of the difference between both implementations (the error) can be zero, thus, it is not statistically significant. However, when the confidence interval is too wide (low precision) or has low values when comparing with the overall result, the difference can be insignificant even if the interval does not contain zero.