# A Methodology for the Design of Application-Specific Cyber-Physical Social Sensing Co-Simulators
## 5 Experimental Validation: Results
### First Experiment (Experiment#1): Results  
Table 11 shows the conclusions of the crowd simulation expert about the quality of the presented co-simulators (the methodological one and \"simulator#1\" and \"simulator#2\").  
As can be seen, the methodological co-simulator is globally better than the non-methodological co-simulators (implemented without following the proposed methodology), and the selected benchmark simulator, for the selected application. Thus, the methodology fulfills the objective of creating the most adequate co-simulator, given a certain application of CPSS. Next, we are reviewing in detail each one of the five quality parameters.  
The expert considered all co-simulators require users to know details about the cyber world, which is not desirable at all. However, in the case of the methodological co-simulator, the third-party engine covers many of the technical details which must be controlled by user in the secondary co-simulators. In relation to this point, nevertheless, MASIS simulator is the best as it includes a graphic interface which is very useful for crowd simulation researchers.  
Similar to the previous discussion, the third-party engine offers a common and easy mechanism to extend the proposed co-simulator if new devices or domains (such as the artificial world) want to be included. On the contrary, the \"simulator#1\" co-simulator requires to design a new middleware each time a new domain-specific co-simulator is included, which is not easy for non-technical users. The same problem appears on \"simulator#2\" co-simulator, as every new device requires creating an equivalent description in MATLAB language, which is most times unknown by social experts. On the other hand, MASIS simulator offers an embedded library of available devices which is very difficult to extend (it requires to initiate a new development and to have technological knowledge).  
In respect to scalability, the expert found the methodological simulator may address every simulation scenario for crowd management systems verification perfectly. This characteristic is also available in the non-methodological \"simulator#2\" co-simulator, as it is an intrinsic property of \"stop-and-wait\" coordination mechanism (as we are seeing later). On the other hand, \"simulator#1\" co-simulator has many problems with certain scenarios, as it must manage a great amount of signalization (typical of choreographed schemes) which hinders the simulation performing. MASIS simulator, however, includes a heavy 3D interface which causes large scale simulations to experiment problems when a high number of elements are simulated (see Section 5.3).  
The models, as the expert said, were adequate in all cases; although some improvements (such as extending the social part) could be included. Experts also indicated that simulation model included into MASSIS shows some deficiencies in respect to mobile entities (for example, they cannot correct the itinerary if a wrong path is followed).  
In general, accuracy in all co-simulators is acceptable. However, non-methodological co-simulators (as well as MASSIS simulator) allow final users to control more exactly this parameter (as the third party engine hides some details), so they are more positively considered. In respect to customization, in general, the non-methodological and MASSIS co-simulators allow users to modify the models, simulation routines, etc. in an easier way than the methodological co-simulator (mainly because of the third-party engine).  
Finally, the results presentation in the methodological co-simulator was very useful for crowd analyses. But the secondary co-simulators need a more extended catalogue of functionalities in this sense (for example, including advanced animations). MASSIS includes a useful graphic interface but certain representations are difficult to obtain.  
In balance, as we said, the proposed methodology allowed implementing the most adequate co-simulator.