## 3 System Implementation and Performance Evaluation
### Platform Selection  
The integration of big data tools aims to achieve the best performance and efficiency of data retrieval. In order to fulfill this goal, the so-called tool selection implements the appropriate tool to finish a job as soon as possible. Therefore, the incoming query command will be dispatched to the appropriate tool to speed up data retrieval. Hive is a tool supported for SQL command that is executed by MapReduce at Hadoop tool. It must write to disk too many times, resulting in low performance of the query task. To address this problem, Impala uses C++ language to re-write its own high-performance Message Passing Interface (MPI) search engine, dramatically reducing hard disk writing and significantly improving performance. Spark implements in-memory MapReduce technology, ensuring that SparkSQL has a very fast processing speed. Although these three tools are functionally similar, their environment requirements and performances differ. This paper has tested a variety of environments that have different memory capacities for the experimental results. When the remaining memory capacity is 2 GB or less on each server, Impala and SparkSQL had a lot of page swapping, causing extremely low performance or crash that will be shown in Section 4 Experimental results and discussion. When the data scale was larger, it caused the JVM I/O exception and made the program crash. However, when the remaining memory capacity was sufficient, SparkSQL was faster than Hive and Impala. Impala's consumption of memory resources was between those of SparkSQL and Hive. This amount of remaining memory was sufficient for Impala's maximum performance. In the experiment as discovered in Section 2.4, each server was allocated 20 GB of memory allocated to every computing node and set the remaining amount of memory to 3 GB as critical point 1 (denoted as L1) and 15 GB as critical point 2 (denoted as L2). The program automatically selected Hive when the remaining amount of memory was less than L1, Impala between L1 to L2, and SparkSQL when larger than L3. In fact, the critical point values vary with the different conditions of the experimental environment. The platform selection algorithm is referred to as PSA, and in PSA, an SQL command is viewed as a user's job forming so-called a job pattern (\(u\), \(V_{C}\), \(E_{C}\), \(f_{v}\), \(f_{e}\)). This paper uses this pattern to develop the automatic platform selection algorithm. Before illustrating the algorithm, we first present the notation it uses. Let \(u\) denote the new user's job and \(u^{\prime}\) the proceeding user's job in U, where U stands for the set of user jobs. \(f_{v}(u)\) denotes the amount of the remaining memory unused in each node (virtual machine) when the job \(u\) is presented. \(f_{e}(u^{\prime})\) denotes how long the rest of execution time to finish the job \(u^{\prime}\) will take, where the predicted execution time for a certain amount of data initially has been made according to the experience and then \(f_{e}(u^{\prime})\) can be estimated by deducting the time taken currently from the predicted execution time. The value of \(f_{e}(u^{\prime})\) could be an integer \(k\) or \(\infty\). \(V_{C}\) denotes nodes \(v_{1}\), \(v_{2}\),...,  
Figure 12: Flowchart of input queries proceeded in multiple big data tools. and \(v_{i}\) in the cluster, and \(E_{C}\) denotes platforms \(e_{1}\), \(e_{2}\),..., and \(e_{j}\) in the cluster. A job pattern is defined as (\(u\), \(V_{C}\), \(E_{C}\), \(f_{o}\), \(f_{e}\)) and an incoming job has been stored in the command list where \(L_{C}\) denotes the command list. Once a job has been assigned to the appropriate platform for the execution, this directly puts it into the job queue. In contrast, if the job has not been assigned to any platform, it is put into the job buffer to wait a while.