# Edge Computing Simulators for IoT System Design: An Analysis of Qualities and Metrics
## 5 Discussion and Research Gaps  
In Section 3, we presented the process of deriving qualities and related metrics for simulating the deployment of IoT systems in edge computing. In order to reduce the risk of human judgment, for the identification of the qualities, we conducted a systematic search to extract the metrics used by other researchers. Then, to rely on a solid base, the ISO/IEC standards were used to classify the identified metrics, however, we found that some of the important identified metrics, i.e., processing, network, and storage delay were not explicitly mentioned by the standards. Nevertheless, they are considered to be finer-grained metrics under mean turnaround time. Moreover, energy consumption was used frequently in the literature, but there is no related metric in the ISO/IEC 25023 standard, which is why we added energy consumption as a metric, however, we do not claim that the derived qualities and metrics are a complete list. Nevertheless, we believe it is an appropriate set and a suitable reference to evaluate the current edge computing simulators.  
Regarding the qualities and metrics supported by the edge computing simulators (as represented in Table 1), our analysis revealed that none of the simulators cover more than five of the 13 identified metrics. With the rapidly increasing amount of research on edge computing, there is still a need for simulators that can assess a wider set of metrics. Among the metrics, delay is the most popular one that is measured by all the existing simulators, but in different ways. Mean response time, processing delay, and network delay are the most common time-behavior metrics supported by the simulators. Among resource utilization metrics, network usage is the most common. Processing utilization and energy consumption were also supported by two simulators, however, we expected more measurements of resource utilization from the current simulators, since efficient resource usage is considered one of the main motivations for the emergence of edge computing. Moreover, some of the identified metrics such as throughput, system availability, and fault avoidance were not supported by any of the simulators.  
It is also worth mentioning that, in this study, we analyzed edge computing simulators in terms of the supported qualities and not regarding features such as mobility and infrastructure modeling. For an analysis of these features, we refer to the survey conducted by [PERSON] et al. [10]. Moreover, the scalability of IoT systems is a key factor that should be supported and evaluated by the edge computing simulators, however, scalability is not considered as a quality by ISO/IEC 25010, and as [PERSON] et al. [32] outlined, a system's ability to scale cannot be analyzed in isolation but has to be assessed with respect to other system qualities, for example, availability or resource usage. To this end, they define scalability analysis as investigating the effect that variations in environmental or design aspects have on qualities of the system. Thus, to facilitate a comprehensive simulation-based scalability analysis of IoT systems and edge computing, different qualities must be supported by the simulator.  
We believe our work can help IoT architects understand the current state of the simulators with respect to supported qualities description and measurement. Moreover, in terms of qualities, we can conclude that there still are gaps. Simulators should cover a wider range of qualities' measurements and modeling new measurements is necessary. Accordingly, we argue that important future work should include the extension of existing simulators to support this, as well as an investigation of the tradeoffs among qualities to provide better support to IoT architects for decision making.