# A Methodology for the Design of Application-Specific Cyber-Physical Social Sensing Co-Simulators
## 4 Experimental Validation: Co-Simulator Development and Experiment Description
### Experiment Description
#### 4.2.2 Detailed Description of Experiments  
An expert on crowd simulations was invited to evaluate the performing and adequacy of the proposed (methodological) co-simulator using a collection of quality parameters. Additionally, the same evaluation was carried out using the non-methodological co-simulators \"simulator#1\" and \"simulator#2\" (described above, Section 4.2.1) and MASSIS simulator. The list of quality parameters was selected to represented how much adequate is the obtained co-simulator for the application scenario. Moreover, the adaptation of the three co-simulators under study to the final users' requirements is also evaluated. These parameters were: usability by crowd management experts, facility to include new types of devices, scalability to advance scenarios, adequacy of the simulation model, accuracy of the simulations, customization and interest of the presented results.  
As can be seen, experiment#1 cannot evaluate which coordination mechanism (parallel execution or \"stop-and-wait\") is the most adequate for our application (mainly because that is a technical decision and depends strongly on the developers being related to the co-simulator's implementation). Then, in the second experiment, a validation was carried out in order to determine if the methodological co-simulator (Section 4.1) implements the most adequate coordination mechanism. For that, the required simulation time by the methodological co-simulator is evaluated, and compared with which required by the non-methodological co-simulator simulator#3.  
Additionally, all results are compared and valuated together with the invested development time in implementing each tool. The validation consisted of the definition of various simulations which were performed using every simulator. Data about the different simulation times were registered. Finally, as we have said in Sections 3.1 and 4.1, every designed co-simulator should fulfill the final users' requirements. However, sometimes, as time passes, the use of the designed co-simulator gets far from what originally planned. In this case, it is very interesting to know if the new tool continuous meeting the final users' requirements in that new context. Then, in the third experiment, three evaluations were carried out in order to determine the operational limits of the tools designed with our methodology. Moreover, result of the experiences described below were compared with results from all the implemented non-methodological co-simulators (\"simulator#1\", \"simulator#2\" and \"simulator#3\").  
In this third experiment, firstly, we are evaluating the scalability of the proposed simulators, depending on the number of agents per simulation. For that, the number of agents in a certain simulation is increased until the designed co-simulator cannot execute it. For every case, various attempts are programmed. Data about the success in the simulations execution are collected. Secondly, the same evaluation is repeated for a fixed number of agents but increasing the number of parameters considered for each agent. The scalability depending on the complexity of the agent model is, thus, evaluated. As previously, for every case, various attempts are programmed. And, thirdly, it is evaluated the accuracy of the resulting co-simulator. For that, the expert on crowd simulations was asked to perform a certain simulation using independent simulation techniques. The results are taken as reference. Then, various simulations modifying the temporal step when possible are performed. Differences between these simulations and the reference (the committed error) are evaluated and employed to determine the simulator's accuracy. As MASSIS describes entities with a fixed number of parameters, and time step is not easily modifiable, this benchmark simulator was only considered for the first part of the experiment. Although the obtained results are not comparable at all (as the complexity of the performed simulations is not similar), these measures may be employed as reference during the discussions.