From: stephen@estragon.uchicago.edu (Stephen P Spackman)
Newsgroups: comp.os.research
Subject: Re: Have we learned anything in the last 20 years?
Date: 6 Apr 92 06:52:57 GMT
Organization: University of Chicago CILS
Originator: osr@ftp

In article <32141@darkstar.ucsc.edu> ken@cs.cornell.edu (Ken Birman) writes:
|I think this whole issue is tightly linked to one's attitude on RPC and on
|asynchronous communication.  If you believe in RPC and are willing to
|pay OS overhead twice on every interaction (once for the message and
|once for the reply, even if the procedure was of type "void"), you 
|probably won't see the merit of asynchronous communication.  If you
|would prefer to see your communication system as a sort of message
|buffer pool, which might collect multiple messages into one packet and
|otherwise amortize costs over many messages, you can drive the effective
|cost of communicate far lower than for RPC.  For example, the current
|commercial version of ISIS can send more than 1000 cbcasts per second
|point to point over a UNIX release from SUN that support about 300 null
|RPC's per second.  The system we are doing at Cornell now will be
|quite a bit faster, but it uses a microkernel approach and won't run
|directly on UNIX at these higher speeds.

But I wouldn't want to say that a compiler/operating system that
understood that void returns or calls that return right into a "select
any three items from this list" operation are special cases would be
NOT implementing RPC. (Why do I feel like this life is a constant
struggle to get work more for compilers? :-). When you compile a
function for use locally you provide pragma settings for speed, size,
safety and so forth, and you give a type signature. It seems eminently
sensible that when compiling an RPC interface you should use the same
information, maybe with a couple of new dimensions. But maybe this is
just semantic....

|A good analogy is with raw file systems (think "RPC") versus buffered file
|systems (think "causal, asynchronous").  In fact, there is even a good
|analogy for the causality issue there: remember when UNIX file systems
|used to get scrambled on every crash?  The problem was basically a
|disk IO subsystem that didn't do physical disk writes in an order
|consistent with causality.  When they added ordering dependencies
|to the buffered IO subsystem in UNIX, it became possible to recover
|file systems automatically after crashes: an illustration that if you
|believe in asynchronous IO, you had better preserve potentially causal
|event orderings.

Except, the user-perceived difference in the two is just that one
works better. So are you really providing an argument against RPC, or
just against naivete in its implementation?

|I tend to agree with the individual who saw RPC going under as networks
|get faster.  Actually, though, I think the key issue is the ratio of
|processor speed to communication latency, or perhaps the processor
|speed compared to the product of latency and throughput (a rough measure
|of how many bytes away your destination is).  The more "distant" your
|destination, the better off you do with asynchronous communication and
|some sort of asynchronous failure notification scheme.  If the destination
|seems close, on the other hand, shared memory makes more sense.

Asynchronous call failure happens also in the local case. Stack
overflow is the usual cause, and you don't want to report it to the
failing thread, leastwise not until after it's recovered (which may be
more than one frame down, since the notification process will need
some stack)....

|I guess this means that in the long run, if current hardware trends
|continue, shared memory may not make much sense at all.  In fact, if
|you look at projects like DASH or MUNIN, with release consistency and 
|explicit mutex primitives wired to the cache, I think that shared memory
|is getting more and more explicitly message-like.  If anything, I would
|suggest that shared memory is basically not a very good idea: it is too
|different from the underlying hardware and hence you can't get adequate
|performance without knowing exactly when messages will get transmitted.

Shared memory is a tiny special case of argument passing: it's what
happens when you pass a pointer to "untyped" memory across the link.
We don't want to eliminate the possiblity, just realise that it's
another case of thinking in assembly language out of habit.
----------------------------------------------------------------------
stephen p spackman         Center for Information and Language Studies
stephen@estragon.uchicago.edu                    University of Chicago
----------------------------------------------------------------------