The Plight of Software Distributed Shared Memory

While hardware shared-memory multiprocessors are enjoying tremendous acceptance and growth in the server market, software distributed shared-memory (DSM) systems have remained confined to a handful of research prototypes. Furthermore, the research in this area has in general failed to address the key shortcomings that have hindered the wider acceptance of software DSM. The goal of this short article is to clearly identify the typical shortcomings of software shared-memory systems, with the hope that this information will serve as a blueprint for a major revision of the research directions in this area. Metrics for Comparison The implicit premise we begin with is that software sharedmemory systems strive to provide a viable alternative to hardware systems. Therefore, the first step towards aligning our research goals to the realities of the marketplace is to use an appropriate set of metrics for comparing software shared-memory systems with hardware systems. While the three traditional metrics of functionality/generality, cost, and performance are well known, they are often not used properly in judging the merits or shortcomings of software shared-memory. Furthermore, to have any relevance to the multiprocessor server market, the above metrics must be judged in the context of leading applications and workloads that represent the largest and fastest growing segments of this market. The following describes the above three metrics along with the appropriate applications to focus on. We next discuss issues related to hybrid hardware/software designs. Finally, we conclude by considering the future for software DSM. 0 Functionality/Generality There is a huge (and growing) rift between the functionality/generality of hardware shared-memory systems and that supported by a typical software system. Furthermore, this is by far the most ignored metric of all, leading to many studies that compare apples to oranges. First, a hardware shared-memory system supports the shared memory functionality at the most basic level. Hence, this functionality is available to all programs (including the operating system) and programming languages. Furthermore, this functionality is uniformly supported for all memory locations and can be used for any purpose (e.g., both for communication and for synchronization). Second, the system as a whole presents the appearance of a single machine, also known as a single-system image. One benefit is that operating system services are transparently available at all processors or nodes. In addition, singlesystem image greatly reduces the complexity and cost of system management. Finally, as a result of commercial market forces, hardware multiprocessor systems are moving towards more robust designs that are suited for mission critical applications. This typically involves some support for fault-containment, and the ability to do hardware/software upgrades with as little downtime as possible. Meanwhile, the typical litmus test for the functionality of a software shared-memory system has been whether it can successfully execute a few Splash-like applications. 1 This has led to a radically different programming model for software systems. First, the shared-memory functionality is only available to a restricted class of programs and programming languages. Typical restrictions include the use of a “fork-based” model for parallelism with the shared-memory functionality limited to heapallocated memory. Second, shared-memory is not supported for all memory locations. Applications typically have to conform to a stylized (and restrictive) use of the shared memory where accesses to memory locations are synchronized to avoid any races. One consequence of this is that the shared-memory functionality cannot be used directly for implementing synchronization. In fact, synchronization is only supported through a limited set of synchronization primitives (e.g., only locks and barri1For most software systems, even porting the Splash applicat ions requires source code changes and can be a non-trivial task! ers) provided by the system (that in turn uses message passing to achieve this functionality! ). Another consequence has been the use of more aggressive and non-standard memory consistency models that typically sacrifice generality and ease-of-use, moving the programming model further and further away from that of a hardware shared-memory system. Finally, there is no resemblance of a “single-system image”. Therefore, real applications with even a modest use of OS services (e.g., resource scheduling, I/O) are not supported. In addition, a system manager is painfully aware of all the nodes as individual computer systems. It is a well-accepted fact that the higher performance and wide-spread availability of hardware systems makes them the platform of choice for application developers. Due to their limited functionality and divergence from standard programming models, software systems cannot directly execute the increasing number of applications being developed for hardware systems and thus fail to capitalize on this growth. In addition, porting such applications to software systems (even if they can cope with the limited functionality) is often not an option because software vendors typically distribute applications in binary format only for a few popular hardware platforms. And even when source is available, porting the application to run correctly and perform reasonably on a software platform is often a daunting task. This problem is exacerbated by the fact that the modifications required for conforming to the programming model and achieving performance on the software platform frequently affect correctness and are not in the form of performance hints. The most viable metric for functionality/generality that can address the above problems is to require software systems to be transparent , i.e., binary compatible with a popular hardware platform. The requirement to execute hardware binaries will not only make a huge number of applications available on software systems, but will also naturally eliminate many of the limitations that makes the development and porting of code difficult on such systems. There are a number of difficult challenges in achieving the transparency goal described above. The only work we are aware of that addresses some of these issues is in the context of Shasta [1]; that paper provides a detailed discussion of typical limitations of software systems and the problems that must be solved on the road to supporting transparency.