## 6 Performance Evaluation  
The IVM-ReLU proposed in this paper conducted a performance evaluation on whether idle virtual machines in OpenStack provide computing services efficiently to the requester for a large-scale job. OpenStack used for performance evaluation is version 3.19.0, which is Stein, the latest release of OpenStack [21], and hardware specifications used as IVM-ReLU Manager are 72 CPU cores, 123 GB of memory size, 200 GB of storage. In addition, the hardware specification of the server where several virtual machines are created includes 32 CPU cores, 134 GB of memory size, and 100 GB of storage; 8 CPU cores, 16 GB of memory size, and 100 GB of storage. The specifications of the virtual machines created inside each server consisted of four types, largely 2 CPU cores, 2 GB of memory size; 4 CPU cores, 4 GB of memory size; 4 CPU cores, 4 CPU cores, 6 GB of memory size; 6 CPU cores, 8 GB of memory size. Among idle virtual machines created within OpenStack, the criteria to be selected as workers were from high-performance idle virtual machines to workers in descending order. The computing service was used as a Sobel Edge Detection algorithm mainly used in image processing, and the task unit for evenly dividing the large-scale job was set to 1 MB. Because 1 MB is the common size of JPEG files generated through smartphones, digital cameras, etc., the performance evaluation designated task unit as 1 MB, considering that the file to be submitted to one of the IVM-ReLU's computing services, the file to be submitted to the Sobel Edge Detection, is JPEG. Figure 10 compares the execution time when requester selects two to ten workers among idle virtual machines to process a large-scale job with different computing size through IVM-ReLU computing service. The specifications of the idle virtual machines used in the performance evaluation consist of 6 CPU cores, and 8 GB of memory size (2 virtual machines); 4 CPU cores, and 6 GB of memory size (2 virtual machines); 4 CPU cores, and 4 GB of memory size (6 virtual machines), and the best specification is selected first. For example, if only two of the idle virtual machines are to be selected as workers, the selected specifications are two idle virtual machines with 6 CPU cores and 8 GB of memory size. As Figure 10 shows, when the computing size is 1 MB, two workers must be used to process the task, resulting in minimal execution time. When the computing size is 72 MB and 154 MB, if requester increase the number of workers to 3, 4, or 5 to process the job, the execution time gradually decreases, but if requester use more than 6 workers, it cannot see much difference. Especially when the computing size is more than 383 MB, there is a big difference in the execution time when selecting 2 workers and when selecting 8 workers. However, the execution time is similar when requester select 9 or 10 workers and when requester select 8 workers. Therefore, it can be seen that there are several workers that are most suitable for processing a large-scale task according to various computing sizes. Figure 11 compares the execution time of a large-scale job with IVM-ReLU applied to OpenStack and a large-scale job with one VM without applying IVM-ReLU to OpenStack. IVM-ReLU processes a large-scale job using up to 10 idle virtual machines, and the execution time for With IVM-ReLU shown in Figure 11 is the average of execution times for 2 to 10 idle virtual machines derived from Figure 10 above. In addition, the specifications of the virtual machine used in the performance evaluation with \"without IVM-ReLU\" is 6 CPU cores and 8 GB of memory size. Comparing \"Without IVM-ReLU\" with \"With IVM-ReLU\", the difference between \"Without IVM-ReLU\" and \"With IVM-ReLU\" is not clear if the computing size is 1 MB, but if the computing size is greater than 72 MB, the gap between \"With IVM-ReLU\" and \"Without IVM-ReLU\" is increased rapidly.