# Edge Computing Simulators for IoT System Design: An Analysis of Qualities and Metrics
## 3 Identification of Relevant Qualities
### Deriving Qualities and Metrics  
Although the ISO/IEC 25010 quality model provides a base to identify the relevant qualities, the model is quite abstract for the domain of edge computing, and a finer-grained level that is related metrics for each quality is required. For this purpose, we used the ISO/IEC 25023 standard [24] that provides a set of measures (metrics) for each quality, however, not all of the metrics in the standard are applicable to edge computing. Thus, to identify the relevant metrics from ISO/IEC 25023, we relied on the extracted metrics from the literature. Before deriving the final set of qualities and the related metrics, we found that some extracted metrics from the literature were not appropriate to be used on generic simulators for edge computing, for example, as they were application-specific or could not be measured by simulators. Accordingly, the following criteria were used to exclude the metrics:  
* Application-specific metrics, such as decision time of an application, ECG waveform, or wireless signal level;
* Not possible to measure by simulation, for example, user experience can be measured by survey;
* Can be calculated by other metrics, such as the average slowdown of an application, that is average system waiting time divided by its service time, can be calculated by combining other metrics;
* Abstract metrics without detailed information regarding how to measure it, such as degree of trust;
* Metrics that are input for the simulation, such as processing capacity of a device.  
The list of the extracted metrics, after applying the filtering, is available in Appendix A, Table A1.  
Finally, by considering the similarities among the extracted metrics from the literature, we identified relevant qualities based on ISO/IEC 25010, and a set of metrics based on the ISO/IEC 25023 standard. We also added four more refined metrics. Three of the refined metrics were identified under mean turnaround time (processing delay, network delay, and storage read/write delay). We found them important metrics, since they are frequently used by the literature. Moreover, energy consumption was another metric frequently used by the literature, but it was not among the metrics in the ISO/IEC 25023 standard. The identified qualities and the related metrics are shown in Figure 3. In the following, a brief description of each metric is provided:  
* **Mean response time:** The mean time taken by the system to respond to a user or system task.
* **Mean turnaround time:** The mean time taken for completion of a job or an asynchronous process.
* **Processing delay:** The amount of time needed for processing a job.
* **Network delay:** The amount of time needed for transmission of a unit of data between the devices.
* **Storage read and write delay:** The amount of time needed to read/write a unit of data from disk or long-term memories.
* **Mean throughput:** The mean number of jobs completed per unit time.
* **Bandwidth utilization:** The proportion of the available bandwidth utilized to perform a given set of tasks.
* **Mean processor utilization:** The amount of processor time used to execute a given set of tasks as compared to the operation time.
* **Mean memory utilization:** The amount of memory used to execute a given set of tasks as compared to the available memory.
* **Mean I/O devices utilization:** The amount of an I/O device busy time that used to perform a given set of tasks as compared to the I/O operation time.
* **Energy consumption:** The amount of energy used to perform a specific operation like data processing, storage or transfer.
* **Transaction processing capacity:** The number of transactions that can be processed per unit time.
* **System availability:** The proportion of the scheduled system operational time that the system is actually available.  
**Fault avoidance:** The proportion of fault patterns that has been brought under control to avoid critical and serious failures.