## 4 Results Evaluation
### Model Validation
#### 4.2.2 Empirical Macro Validation  
Empirical macro validation of the model's output includes showing that the output of the implemented model corresponds to the real world [88]. This is a key test for the model's validity. Empirical output validation tests the model designer's hypothesis (i.e., the implemented model). There are three different ways to empirically validate a model's output: _Real-world data_, _stylized facts_, _and cross-validation_. Real-world data validation is crucial if the model is going to be used as a predictive model. In this case, there may often be only one real historical data set, and many runs of the model; thus, empirical output validation includes showing that the real world is a possible output of this model (i.e., that the real-world dataset lies within the statistical distribution of the model data). If there are many outputs from the model and the real world, then real-world data validation includes showing that the average model results correlate with the real-world results [57; 94]. A stylized fact is a general concept of a complex system derived from the knowledge of domain experts. If the model's major purpose is to be a used as a thought experiment, then validating the model against a stylized fact is enough to show the validity of the model [90]. Cross-validation (or docking) is an optional validation process that compares the new model against another model that has already been validated, even if that other model uses another methodology (e.g., EBMs such as system dynamics). If the two models generate similar results, then the validity of the new model has been enhanced [92]. In this study, the outputs of the model were empirically validated against real-world datasets ranging from the beginning of summer 2017 to the end of autumn 2017 (i.e., six months). Since the model is a multi-output model, three various datasets (2G, 3G, and 4G) have been used for its output validation. These datasets were provided by ITRC and are visualized in Appendix B. Empirical validation of a model's outputs includes three parts of (1) number of experiments, (2) data normality testing, and (3) correlation coefficient measurement. In case of empirical outputs validation, it should be noted that if its results don't show an acceptable level of accuracy in matching up with real-word data, the model' elements, such as properties, input parameters, and even behavioral rules, have to be re-calibrated. The process of model's output empirical validation is visualized as Figure 9.  
Number of Experiments  
Since ABMs have a stochastic nature, they produce various outputs, even if they have one specified combination of input parameters. So, the statistical distribution of the model's multiple experiments (i.e., runs) would be considered essential in determining its correspondence with real-world data [52, 78, 90]. The presented model in this study is a three-output model where the diffusion of three competing telecoms technologies (i.e., 2G, 3G, and 4G) is simultaneously simulated while there were two mutually exclusive social networks in which the combination of numeric values (As provided in Table 8) should be tested. For this purpose, using _BehaviorSpace_ tool, the model was randomly run by 1500 experiments for each network type, which resulted in 1500 experiments for the performance of every MTT in each of social networks (i.e., totally 900 experiments with positive random seeds ranging from 1 to 1500) as presented in Table 9. This has majorly helped better understand which type of social network fits that of our case study. Figure 9: Steps of output empirical validation and its impact on model’s elements re-calibration. Data Normality Testing  
To get a deeper insight concerning the predictive accuracy of the model's outputs, the correlation coefficient test should be implemented. To find that which correlation test has to be used, knowing the normality of the data (both model's outputs and real-world data) is necessary.