# Constructing Reliable Computing Environments on Top of Amazon EC2 Spot Instances +
## 4 Evaluation Scenario
### Performance Metrics  
In order to analyze the proposed solution, we defined four metrics to score the performance of the scheduling algorithm and container-based migration strategy to build reliable virtual cluster execution environments. The first metric is shown by Equation (9) and denotes the task efficiency \(E_{T}\), which is computed as the ratio of the number of successfully finished tasks \(T_{F}\) to the total number of tasks submitted \(T_{S}\).  
\[E_{T}=\frac{T_{F}}{T_{S}}, \tag{9}\]  
The second metric, workflow efficiency \(E_{W}\), is presented in Equation (10). It is determined as the ratio of successfully completed workflows \(W_{F}\) to the number of submitted workflows \(W_{S}\).  
\[E_{W}=\frac{W_{F}}{W_{S}}, \tag{10}\]  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**VM Instance** & **O (USD/hour)** & **S (USD/hour)** \\ \hline t2.small & 0.0230 & 0.0069 \\ t2.medium & 0.0464 & 0.0139 \\ t2.xlarge & 0.1856 & 0.0557 \\ t2.2 xlarge & 0.3712 & 0.1114 \\ \hline \hline \end{tabular}
\end{table}
Table 1: On-demand (O) and spot instance (S) characterization price per hour.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**VM Instance** & **B 1 h (USD/hour)** & **B 6 h (USD/hour)** \\ \hline t2.small & 0.0130 & 0.0160 \\ t2.medium & 0.0260 & 0.0320 \\ t2.xlarge & 0.1020 & 0.1300 \\ t2.2 xlarge & 0.2040 & 0.2600 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Spot block instance (B) characterization price per block duration.  
The third metric, planning efficiency \(E_{P}\), is expressed by Equation (11), and it is calculated as the ratio of number of tasks \(T_{R}\) successfully scheduled to the total number of tasks submitted \(T_{S}\). \(E_{P}\) is determined for the whole system.  
\[E_{P}=\frac{T_{R}}{T_{S}}, \tag{11}\]  
The fourth metric \(E_{C}\) intends to measure the performance of the scheduling algorithm in terms of both monetary costs and average task runtime. \(E_{C}\) is expressed by Equation (12). For a particular experiment \(i\), \(E_{C}\) is determined in two steps: (1) it first calculates \(C(x_{i})\), which is the ratio of the task efficiency \(E_{T}(x_{i})\) to the product of average task runtime \(AVG\_TR(x_{i})\) and total monetary cost \(MCost(x_{i})\) with rented time slots found necessary to execute the tasks belonging to each workflow; (2) and then, this result is normalized with reference to the maximum result obtained, \(max\{C(x_{j}):j=1,\,\ldots,n\}\), considering all the \(n\) experiments, in order to express the result in the range \([0,1]\). \(E_{C}\) increases as: (1) the number of successfully finished tasks increases; (2) the average task execution time decreases; and (3) the monetary costs in executing tasks decreases.  
\[\begin{split} E_{C}=\frac{C(x_{i})}{max\{C(x_{j}):j=1,\ldots,n\}},\\ \text{where }C(x_{i})=\frac{E_{T}(x_{i})}{AVG\_TR(x_{i})\times MCost (x_{i})}\end{split} \tag{12}\]  
The rationale behind \(E_{C}\) is to measure how well the scheduling algorithm performs in providing a time and cost-effective service to end user. Both costs and runtimes are considered since none of the tested scheduling algorithms deploys the means to choose which parameter dominates the other (i.e., if runtime is more important than cost, or vice versa). Apart from the ratio of successfully finished tasks \(E_{T}(x_{i})\), the quality of the service provided also depends on the \(AVG\_TR(x_{i})\times MCost(x_{i})\), which expresses the concept of service cost perceived by end users. By means of an example, the service cost tends to increase as the average task runtime and/or monetary costs related to instances renting are augmented.  
All the metrics fall into the \([0,1]\) interval, and the performance of the scheduling algorithm increases as the metrics approach the unity.