School of Information and Computer Science Georgia Institute of Technology

In recent years there has been increasing interest in message-based operating systems, particularly in distributed environments. Such systems consist of a small message-passing kernel supporting a collection of system server processes tha t provide such services as resource management, file service, and global communications. For such an architecture to be practical, i t is essential tha t basic messages be fast, since they often replace what would be a simple procedure call or "kernel call" in a more traditional system. Careful study of several operating systems shows tha t the limiting factor, especially for small messages, is typically not network bandwidth but processing overhead. Therefore, we propose using a special-purpose coprocessor to support message passing. Our research has two parts: First, we parti t ioned an actual message-based operating system into communication and computation parts interacting through shared queues and measured its performance on a multiprocessor. Second, we designed hardware support in the form of a special-purpose smart bus and smart shared memory and demonstrated the benefits of these components through analytical modeling using Generalized Timed Petri Nets. Our analysis shows good agreement with the experimental results and indicates tha t substantial benefits may be obtained from both the parti t ioning of the software and the addition of a small amount of special-purpose hardware. 1. I n t r o d u c t i o n We are interested in providing hardware support to improve the performance of distributed systems. For the purposes of this paper, a distributed system is defined as a collection of computing nodes interconnected by a local area network (LAN). There are one or more processors and a certain amount of memory in each node. The nodes do not share memory; message exchange across the network is the only mechanism for communication between them. Many recent operating systems designs for such an environment [Baske 77, Cheri 83, Finke 83, IBM 83a, Kepec 84, Lee 84, Rashi 81, Solom 79, Tanen 81] place a message-passing kernel on each node, supporting processes and communication between them via explicit messages. This kernel supports both local communication--communication between processes on the same node---and non-local or remote communication (sometimes implemented via a distinguished network manager process). 1This work was supported in part by the Defense Advanced Research Pro. jects Agency, the National Science Foundation, and International Business Machines Permission to copy without fee all or part of this material is granted provided that the copies are not made or distributed for direct commercial advantage, the ACM copyright notice and the title of the publication and its date appear, and notice is given that copying is by permission of the Association for Computing Machinery. To copy otherwise, or to republish, requires a fee and/or specific permission. Access to system services are requested via protected procedure calls in a traditional system, whereas in a messagebased operating system they are requested via message passing. While a simple procedure call costs jus t a few instructions, and a protected procedure call (kernel call) costs a few hundred instructions, IPC costs a few thousand instructions in several systems tha t we studied [Ramac 86]. Since message exchange is the basic kernel mechanism in message-based operating systems, the performance of the system depends crucially on the rate of message exchange. Our measurements (see § 3) as well as the measurements of others [Artsy 84, Cheri 83, Gagli 85] indicate tha t for small messages (which make up the vast majority of all messages sent [Cheri 83], the limiting factor is the high processing overhead tha t is incurred in message passing ra ther than limited network bandwidth or the time to copy messages from buffer to buffer. There are two important figures of merit in this environment: round-trip time, and message throughput. Round-trip time is the elapsed time seen by an application between sending a message and receiving a reply from the intended receiver. This figure of merit affects an individual application's performance. Message throughput is a global figure of meri t t ha t determines the performance of the entire system. Informally, i t is the number of messages tha t the system is handles per uni t time. We show tha t a modest amount of additional hardware can significantly improve message throughput and average round-trip t ime in a mult iprogramming environment. We also show tha t additional hardware support in the form of high-level bus primitives affords even greater improvement in communication subsystem performance. Available hardware support [ABLE 84, Inter 83] and previous modeling studies [Gorad 87, Woods 84], address the issue of off-loading communication protocols onto front-end processors, and provide evidence tha t this approach can have a significant performance payoff. However, these and other previous proposals of hardware support for interprocess communication (see survey in [Ramac 86]) are more limited tha t the study reported in this paper in several respects. First, previous work generally assumes "communication" to be the work tha t is performed by the operating system to satisfy non-local requests. However, for message-based operating systems, measurements by ourselves [Ramac 86] and others [Cheri 83] show tha t there is a high processing overhead for local communication as well. Second, many proposals include only limited, low-level support for communication, leaving out support for operations such as address translation, control block manipulation, and kernel buffering, which account for a substantial portion of message passing overhead. Finally, a front-end processor for a specific network protocol (such as TCP/IP [Posts 81]) may not mesh well with the operating system primitives, and therefore may incur higher overhead than necessary. The problem needs to be addressed at a much higher level. 2. Goals Our goal is to improve message throughput by providing concurrent processing support for message passing. The organization we propose inside each node is shown in Figure 1. There is a host in each node tha t executes the message-based operat178 © 1987 ACM 0084-7495/87/0600-0178500.75