## 3 System Architecture and Implementation
### Adaptive Resource Allocation Policy  
We collected hardware specifications of the host server to set the _maximal_ resource limits of container running environments in terms of CPU and memory by using '/proc/cpuinfo' and the 'free' command, respectively. As we mentioned before in Section 3.1, for continuous resource usage monitoring, we utilized cAdvisor [28] developed by Google. However, there is another method for monitoring Docker containers by using the 'docker stats' command [11]. Compared to cAdvisor, it can be more intuitive and simpler to use; however, according to our experience, the information on container resource usages may not be accurate. In addition, sometimes, it takes relatively long time to fetch the necessary information due to the communication with containers that make it an unreliable approach for employing agile vertical elasticity. Automating the vertical elasticity of Big Data workloads requires adaptive resource allocation based on checking container resource information at accurate intervals, so that monitoring through cAdvisor is adopted in our proposed scheme. cAdvisor can be run as a process or a Docker container, and it records resource usages of the host server and Docker containers. These collected monitoring data can be fetched through REST APIs, and the web UI for users is also provided. Some information obtained through cAdvisor are presented in Table 1.  
Big Data workloads typically require large amounts of CPU and memory, and the resource usage widely fluctuates depending on the idle time when a new job is allocated or a next job is dispatched. In order to distribute and process such Big Data workloads in the cloud infrastructure, efficient resource allocation is required. For this purpose, we designed and implemented an adaptive resource allocation technology that can optimize the overall system utilization of Big Data workloads. Specifically, the monitoring cycle of Adaptive Resource Controller (in Figure 1) is set to 3 s, which can accelerate the overall resource adjustments (i.e., scaling up/down) compared to other approaches [13]. In addition, CPU and memory revision policies are devised to suit the Big Data workloads. When scaling allocated computing resources, CPU and memory are managed _separately_, and several criteria are developed according to the resource usage patterns. Therefore, a larger amount of computing resources can be adjusted at once, which can improve the aggregated system utilization and throughput. The detailed container resource adjustment and monitoring policies that are empirically determined by reflecting the characteristics of Big Data workloads are presented in Table 2. For this purpose, we performed extensive profiling and experiments to investigate resource usage patterns (in terms of CPU and memory) of various Big Data workloads from the HiBench suite and determined the thresholds and scaling metrics presented in Table 2. Specifically, we observed that the CPU occupancy was allocated and returned by Docker containers with ease, so that thresholds for CPU coordi \begin{table}
\begin{tabular}{c c} \hline \hline
**Items** & **Description** \\ \hline docker/aliases & Name of docker container \\ spec/memory.limit & Memory limit of the docker container \\ docker/memory.usage & Memory usage of the docker container \\ stats/cpu.usage.total & CPU usage of the docker container \\ stats/timestamp & Monitoring time \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Information provided by cAdvisor that can be utilized for monitoring Docker containers. lenient. On the other hand, the allocated memory occupancy for a Docker container was not easily returned mainly due to the use of cache memory, so that thresholds for memory are made a little tighter. Therefore, in the case of memory management, it was set to be tight in order to prevent a single container from monopolizing the main memory usages since most Big Data workloads failed with insufficient memory allocations. In the case of CPU, it was set to be allocated and returned more leisurely.