From: bellcore!gizmo.bellcore.com!mo@uunet.UU.NET (Michael O'Dell)
Subject: Problems with the word "distributed"
Organization: Center for Chaotic Repeatabilty
Date: Wed, 15 Apr 1992 21:48:55 GMT

Here beginneth the screed:

One of the problems I see with "distributed systems research" is that
the word "distributed" means far too many things to possibly ever work.

I fear "transparent distribution" is not an oxymoron only for small,
well-chosen values of "distributed."

To pick on one common approach to transparent distribution at random:

"distributed virtual memory" = DVM

constructs a facade of non-remoteness on memory load/store boundaries
using varying size cache lines, blocks, or pages, depending on the
implementation, and some version of a cache-coherency protocol
for suffling the shared objects back and forth as needed.

DVM seemingly works fine on small shared memory multiprocessors, 
but it starts to groan a bit on a large bus-based NUMA machine,
and it becomes more tenuous in a building of workstations,
becomes genuinely questionable over a large campus,
and reduces to a joke when used across the continent where
speed-of-light delays reduce transaction rates to 30/second.
Let's not even mention what happens when you want parts of
your distributed system in low earth orbit, geosynchronous orbit,
on the Moon, or on a spacecraft meeting Saturn.

So, I conclude that DVM is not a suitable primitive for distributed systems
because it does not scale.   I think it was Dave Clark who once said
that if your architecture doesn't scale gracefully in the face of 
increasing delay, you have in fact built a shared-memory multiprocessor,
no mater how you actually wired it up.

I see are two ways out of the DVM dilemma: define "distributed"
in such a way that DVM still works, and think of a new word for large,
complex systems which run on multiple processors and computers spread
over potentially large areas, or, rethink the fundamental approach
to "distribution."  I can't prove it yet, but my gut feeling is that
the "remote execution" paradigm is the closest thing I've yet seen
to something that might scale up.  I think it scales down because
the limit is just a fixed RPC-style transaction, while the upper
limits let one throw an arbitrary and VARIABLE amount of work
per transaction without having to recompile any "remote server" code.
This allows dynamic adaptation and very, very late (even lazy) binding
so transaction "throw weight" can be tailored on a per-interaction basis.

There is yet another less pleasant alternative:  we might discover that the
world is hopelessly heterogeneous and one simply can't fashion one mechanism
which can scale at least 6 if not 9 orders of magnitude in delay, and hence we
MUST deal with partitioning and mechanism choice at the applications
design level.  Note that I believe the last half of the previous statement
no matter whether remote execution can cut it or not. Thinking about
partitioning and protocols is a fundamental part of design and I do
not believe, as a matter of faith, that quality software, especially
distributed system software, can be built without careful attention to
these details, for the Devil dwells therein.

So, end of screed.

	-Mike O'Dell
	 Resident Crank, turning the crank one more time

Bellcore? Bellcore isn't allowed to have opinions.  These be mine.