## 4 Performance Evaluation
### Experimental Design  
This study focuses on reducing remote memory access by improving inter-node load balancing. A sufficient number of benchmark programs must be run simultaneously on the experimental system, such that the kernel scheduler migrates tasks between nodes to balance the load when the nodes have an imbalanced load. 29 benchmark programs with a specific number of threads are run simultaneously for each test case. The numbers of threads range from 1, 2, 4, 8, 16, and 32. The elapsed running time for each test case and the elapsed running time for each benchmark program are measured using the performance counter statistics. For each test case, eight to ten runs are performed.  
For each run, the system is rebooted to prevent buffer caching. During each run, the performance counter events in Table 5 are also recorded for each benchmark program. These are used to determine the cause of any change in performance: Instructions Per Cycle (IPC), Last Level Cache (LLC), and Miss Per Kilo Instructions (MPKI) to estimate runtime patterns. Other performance data is obtained from _numstat_ command-line utility and _vmstat_ in the _proc_ file system.  
This study also enhances existing task selection policies as described in Section 2.3 for multithreaded applications, so the experiments use different task selection policies, as shown in Table 6. The First-Fit policy is used in the unmodified Linux kernel. As presented in Section 3.1, the MB [4] policy considers the memory occupied by one task  
\begin{table}
\begin{tabular}{c c} \hline \hline
**System** & **Supermicro A + 4042G-72 RF4 [18] with 4 NUMA Nodes** \\ \hline Processor & 4 AMD Opteron 6320 8-core Processor, 2.80 GHz \\ \hline Motherboard & Supermicro H8 QG7-LN4F \\ \hline Memory controller & 2 per node \\ \hline Interconnect & 6.4 GT/s AMD HyperTransport \\ \hline \multirow{6}{*}{Cache sizes per processor} & L1 Data cache: 16 KB per core \\  & L1 Instruction cache: 64 KB per 2-core \\  & L2 cache: 2 MB per 2-core \\  & L3 cache: 8 MB among all cores per memory controller \\ \hline Memory & DDR3 1600 16 GB per node (64 GB total) \\ \hline Operating System & Ubuntu 13.10 Server Edition (Linux kernel 3.11.0-12-generic [20]) \\ \hline \hline \end{tabular}
\end{table}
Table 4: Software and hardware specifications.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Event** & **Description** \\ \hline Elapsed Time & Used to evaluate the efficiency of running one benchmark program \\ \hline CPU Cycles & Used to calculate the number of Instructions Per Cycle (IPC) \\ \hline Instructions & Used to calculate IPC and Last Level Cache (LLC) Miss Per Kilo Instructions (MPKI) \\ \hline LLC-load-misses & Used to calculate LLC MPKI. LLC-store-misses is not supported. \\ \hline Page Faults & Number of page faults incurred by threads of one thread group \\ \hline CPU Migrations & Number of task migrations for one thread group \\ \hline \hline \end{tabular}
\end{table}
Table 5: Collected performance counter events.  
on the destination and source nodes to select a target task for inter-node migration. This policy is suited to multithreaded applications. However, additional overhead is incurred because the policy relies on the information provided by the kMLB mechanism [4].  
To allow faster selection of tasks, we also enhance the MB policy to be MB\({}^{+}\). Each task in the runqueue is evaluated only when its thread group leader is currently not located on the destination or the source nodes. Therefore, only the subset of tasks in the runqueue must be evaluated. Besides, a thread group leader is not migrated across nodes, and a task is migrated directly if its thread group leader is on the destination node.