# A Methodology for the Design of Application-Specific Cyber-Physical Social Sensing Co-Simulators
## 5 Experimental Validation: Results
### Third Experiment (Experiment#3): Results  
Once methodology performance has been evaluated, it is important to determine the operational limits of the co-simulator. Figure 18 shows the results about the scalability depending on the number of agents per simulation scenario. As can be seen, every simulation including 5000 agents, or fewer, is perfectly executed by the methodological co-simulator. However, as this number goes up, the simulation algorithm start being collapsed and, in some occasions, simulations cannot be performed. In particular, for a number of agents \(n=\) 50,000 the simulation returns an execution error in 50% of times. For this work, we are considering a simulation fails (or returns an execution error) when the management entity is not able to communicate with at least one of the domain-specific simulators that make up the general co-simulator. Besides, simulations are considered blocked if the updating process in one of the domain-specific simulators does not finish before one Linux _keepalive_ probe (employed, for example, also in TCP connections -75 s). In fact, problems in the simulation may appear due to hardware limitations, software coordination malfunctions or communication stack overflows because of the great amount of time required to perform the simulation update at each time step when the number of agents is above a certain limit. Furthermore, the maximum number of agents that may be considered without appearing  
Figure 17: Function \(S(n)\). problems in the simulation also depends on the machine hosting the simulation environment. In this case, we have selected a 64-bit Linux Ubuntu 16 operating system, with an Intel i5 processor and 8 GB of RAM memory. As this hardware configuration it is not specifically designed to support heavy simulations, obtained results may show a quite small maximum number of agents per simulation. However, as the objective of this experiment is to compare the performance of different co-simulator configurations, obtained measures can be considered adequate. As can be seen in Figure 18, for \(n=80,000\) or more agents the simulation algorithm never worked (in the sense expressed above). The origin of this limit is the third party engine. As this element acts as an orchestrator, all information must be processed by it. Thus, it creates a bottleneck which blocks the simulation, even when the underlying simulators may perform simulations including more agents. This phenomenon is common to all orchestrated schemes, including the non-methodological co-simulators \"simulator#2\" and \"simulator#3\". In the case of these simulators the problem aggravates. Fist, in \"simulator#2\" the orchestrator is one of the domain-specific simulators (MATLAB in this case), so this element not only must perform the orchestration activities but also the simulation under evaluation. Then, the bottleneck narrows and the maximum number of agents decreases (in this case for \(n=7000\) agents the simulations return an error in the 50% of cases). On the other hand, \"simulator#3\" presents a parallel execution scheme. Thus, information is not generated in an ordered way (as in stop-and-wait scheme) and the third-party engine gets overloaded much sooner. This case is the most critical, as can be seen in Figure 18, and for \(n=5000\) agents the simulations fail in the 60% of cases. This situation is relieved if choreographed simulations are considered, as in \"simulator#1\". It presents the widest operation regime as it presents a distributed management system. In particular, this simulator gets blocked when the signaling load among the domain-specific simulators cannot be managed. This limit is reached, approximately, for \(n=70,000\) agents, when simulations return an error the 40% of times. Although these numbers may seem low, greatest public facilities may host between 100 and 200 thousand people. Furthermore, the proposed simulation scenario, as maximum, may host around three thousand people (this quantity is supported without problems by the proposed co-simulator).