# Resource Profiling and Performance Modeling for Distributed Scientific Computing Environments
## 4 Evaluation
### Microbenchmark Experiments  
We also conducted microbenchmark experiments from MTC community [40; 41] on the Biomed VO [12] which is a large-scale international and interdisciplinary virtual organization providing tens of thousands of CPU cores to scientific communities. We used three different types of BoT (Bag of Tasks) jobs to evaluate our forecasting model (**AFMV**) including \"Short\" (sleep 5 tasks), \"Medium\" (sleep 5 tasks) and \"Long\" (sleep 100 tasks) jobs especially from the perspective of Many-Task Computing (MTC) applications [4]. The number of tasks in each BoT job category ranges from 250 to 5000. In addition, four sets of computing elements (CEs) from Biomed VO with different resource characteristics were selected as followings:  
* **Profiled Top 10 CE with Core**: Top 10 CEs based on model predicted maximum available free CPU cores (i.e., best 10 CEs based on our proposed models that are expected to show maximum number of free CPU cores)
* **Profiled Top 10 CE with Time**: Top 10 CEs based on model predicted minimum response time (i.e., best 10 CEs based on our proposed model that are expected to show minimum response times)
* **Grid Info Top 10 CE**: Top 10 CEs based on the number of free CPU cores provided by the Grid Information Service which is an existing conventional monitoring service in VOs. * **Random 10 CE**: Randomly selected 10 CEs (i.e., simply a collection of 10 CEs that are randomly selected)  
Performance metrics are _Makspan_ and _Success Rate_. The makspan of a BoT job is defined as the _time_ to complete the execution of all tasks. Success Rate is defined as the _ratio_ of the number of tasks executed successfully to the number of submitted tasks. Figure 7 shows the Makespans of different sizes of short running BoT jobs with four sets of CEs. Compared to the other three lists of CEs, CEs with the predicted minimum response time (\"Profiled Top 10 CE with Time\") can achieve the shortest Makspan. As the number of tasks increases, Makspans of \"Grid Info Top 10 CE\" and \"Random 10 CE\" grow rapidly. Interestingly, from these results, we can observe that for short running tasks, expected minimum response time CEs (\"Profiled Top 10 CE with Time\") perform better even the \"Profiled Top 10 CE with Core\" potentially have more free CPU cores. Figures 8 and 9 illustrate the Makespans of medium and long running BoTs respectively as we increase the number of tasks. It can be noted from Figure 8 that both predicted top 10 CEs (\"Profiled Top 10 CE with Time\" and \"Profiled Top 10 CE with Core\") completed the execution of BoT jobs comparatively within very short time ranging from 100 to 600 s for 250 to 5000 tasks respectively. On the other hand, the job completion time of randomly selected CEs takes about 1600 s for 5000 tasks which is approximately 3 times more than that of profiled CEs based on our proposed model. For the long running BoT jobs, both predicted CEs lists show almost identical performance, however for 5000 tasks, CEs having maximum number of free CPU cores exhibits slightly better turnaround time. From this, it can be arguable that CEs having more free cores can perform better for a large number of tasks having relatively long running times. Another important criterion for executing HTC/MTC applications on top of a VO is the _Job Completion Rate_ which refers to the percentage of tasks that can be successfully completed. Figure 10 shows that regardless of task execution times (such as Short, Medium, and Long), both of our predicted Top 10 CEs (\"Profiled Top 10 CE with Core\" and \"Profiled Top 10 CE with Time\") can achieve significantly higher job completion rates than the others. Specifically, both of predicted Top 10 CEs show almost 100% of success rates while \"Grid Info Top 10 CE\" hovers around 50% to 70%, and \"Random 10 CE\" shows even worse ranging from 20% to 40%.