A Case for User-Level Page Migration

This paper presents user-level dynamic page migration, a runtime technique which t ransparent ly enables parallel programs to tune their memory performance on dis t r ibuted shared memory multiprocessors, with feedback obtained from dynamic monitoring of memory activity. Our technique exploits the i terative nature of parallel programs and information available to the program both at compile t ime and at runt ime in order to improve the accuracy and the timeliness of page migrations, as well as amortize bet ter the overhead, compared to page migration engines implemented in the operat ing system. We present an adapt ive page migration algorithm based on a competit ive and a predictive criterion. The competi t ive criterion is used to correct poor page placement decisions of the operating system, while the predictive criterion makes the algorithm responsive to scheduling events tha t necessitate immediate page migrations, such as preemptions and migrations of threads. We also present a new technique for preventing page pingpong and a mechanism for monitoring the performance of page migration algorithms at runtime and tuning their sensitive parameters accordingly. Our experimental evidence on a SGI Origin2000 shows tha t unmodified OpenMP codes linked with our runtime system for dynamic page migrat ion are effectively immune to the page placement s trategy of the operat ing system and the associated problems with da ta locality. Furthermore, our runtime system achieves solid performance improvements compared to the IRIX 6.5. 5 page migration engine, for single parallel OpenMP codes and mul t iprogrammed workloads. 1. I N T R O D U C T I O N The current t rends in parallel computing indicate tha t shared memory multiprocessor architectures converge to a common model in which mult iple single-processor or symmetric multiprocessor (SMP) nodes are interconnected via a Permission to make digital or hard copies of all or part of tiffs work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage amt that copies bear this notice and the full citation on the first page. To copy othe~'ise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. 1CS 2000 Santa Fe New Mexico USA Copyright ACM 2000 1-58 l 13 -270-0/00/5 . $5.00 fast network and equipped with addit ional hardware support to provide the abstract ion of shared memory to the programmer [8]. This architectural model was adopted in earlier research prototypes such as the Stanford DASH and FLASH multiprocessors [14] and later, in high-end commercial products including the Sequent Sting [17], the HP/Convex Exemplar [4], the SGI Origin2000 [15] and the Sun Wildfire [10]. Although these architectures promise t o meet the requirements of scalability and programmabili ty, they also present the challenging problem of the Non-Uniformity of Memory Access latencies (NUMA). NUMA introduces the notion of distance between a thread of a parallel program and the da ta tha t this thread accesses. A thread running on one node of the system may access d a t a located in the memory of another node. These memory accesses, albeit t ransparent to the programmer, have significantly higher latency than accesses to local memory and const i tute one of the main sources of performance degradation. Dealing with the latency of remote memory accesses in software poses a tradeoff with respect to the simplicity of parallel programming models for shared memory multiprocessors. Ideally, the main system software components, namely the compiler, the runtime system and the operat ing system should cope with remote memory accesses by applying program transformations and runt ime techniques tO collocate each thread wi th the da ta tha t ~ the thread accesses more frequently. However, despite the effort spent in optimizing system software for NUMA multiprocessors, experiences with real applications make evident that significant programming effort is still required in order to tune data locality and achieve acceptable scalability on moderate and large-scale systems [12]. In particular, the programmer must be aware of the operating system's page placement policy and either modify the program to adapt its memory access pattern to the operating system's policy, or bypass the operating system and hardcode in the program a customized page placement scheme. Both approaches compromise the simplicity of the shared memory programming model, in terms of the effort needed to optimize a program and the subtleties of the programming constructs which are required to tune data placement. The problems related to page placement and locality have already introduced major considerations with respect to the widely popular OpenMP shared memory programming standard and raised the dilemma whether data distribution directives should beintroduced in OpenMP or n o t [16].