## 3 Proposed Cost-Effective Auto-Scaling (CEAS) Framework
### Data Processing and Clustering with Amazon EMR  
The Amazon EMR enables the processing of large data and the workflow of the data processing can benefit from virtual Hadoop clusters [15; 25]. When the Amazon EMR cluster is provisioned, it is crucial to choose the optimal number of instances and their sizes since certain workloads may be disk-I/O or memory intensive when the others are CPU intensive. In the proposed CEAS framework, for comparison between the fixed cluster and auto-scaling cluster, four kinds of Hadoop cluster are considered in the AWS cloud environment with the following settings:  
1. _Three-slave cluster_: 3 slaves with 3 large instances;
2. _Seven-slave cluster_: 7 slaves with 7 medium instances;
3. _Fourteen-slave cluster_: 14 slaves with 14 medium instances;
4. _Auto-scaling cluster_: a different number of slaves which varies from 3 to 15 slaves with the corresponding instances. Intuitively, the fourteen-slave cluster should provide the best performance when processing the big data. However, such employment of all available slaves may be not necessary when the Hadoop only uses the memory as per the requirement of data processing. Also, considering the scenario when memory is not sufficient enough to perform the sorting on large data sets, if certain part of the data is written to disk which is expensive and may slow down the job, then using the instances with high memory is preferred to achieve a better performance. With the Amazon EMR, various instance sizes can be implemented by either switching among various instance profiles or terminating the current cluster and starting a new cluster with different instance size. The demands for both CPU and memory-intensive jobs can be further met by mixing various types of Amazon EC2 instances and by adding or removing the cluster nodes. When provisioning the cluster, depending on the size of the instance, the Amazon EMR can set up various optional settings for the Hadoop configuration, such as the size of the Java memory used per daemon, and the number of mappers and reducers configured per instance. To quickly process the data on Hadoop cluster, the required data nodes need to be added to the cluster based on the data size. Hadoop can process the data by splitting the input files into blocks for parallel processing. Moreover, Hadoop can create more data blocks and tasks to analyse the massive data. The opted Amazon EMR cloud can therefore have only one cluster but with several nodes that can analyse and process the map reducing jobs in parallel. For instance, if a dataset needs 30 mappers to process the input data file consisting of 30 splits realised by Hadoop, then an ideal Amazon EMR cluster would be capable to manage all mappers running in parallel by employing ten small instances each of which can execute three map tasks concurrently. Although parallel data processing is shown to be advantageous, the usage of all mappers is actually not necessary when there is no time constraint and the excessive cost for setting EMR Hadoop cluster is also a major concern. Instead, we can reduce the number of mappers for latency-tolerant jobs. However, the data stream on the HDFS system is distributed across the slave nodes, and thus the scale-down operation on the clusters may switch off a node causing the data loss [11; 12; 25]. In Hadoop, decommissioning was developed to avoid such data loss by transferring the data from an instance to other machines prior to removing that instance. The time consumption for such decommissioning depends on the data size, but this is not tolerable to perform an adequate scaling. Hence, auto-scaling has been proposed to timely collect the streaming data load. By using the Covering HDFS in Hadoop, the cluster can scale down the instances without suffering any data loss [13]. A typical Hadoop cluster consists of [20]: (i) _Core slaves_: to provide both computational power and storage services, and (ii) _Compute/task slaves_: to offer only computational power services.