From: lowry@watson.ibm.com (Andy Lowry)
Newsgroups: comp.os.research
Subject: Re: Have we learned anything in the last 20 years?
Date: 6 Apr 92 10:13:03 GMT
Article-I.D.: darkstar.32421
Organization: IBM T.J. Watson Research Center
Originator: osr@ftp

In article <32014@darkstar.ucsc.edu> ast@cs.vu.nl (Andy Tanenbaum) writes:
   GENERALLY ACCEPTED AS TRUE BY RESEARCHERS IN DISTRIBUTED SYSTEMS
      - The client-server paradigm is a good one

It's useful if it fits your application.  If your application's
communication patterns don't fit the client/server model, it becomes
an obstacle.  Much better to base your systems on a process dynamics
and communication model which encompasses client/server as a special
case.  I need to be able to write programs that can accept calls from
other processes, but nevertheless have their own thread of control,
can call other processes to get services, can create new processes and
reconfigure communications channels, and can decide when and from whom
to accept calls for services.  This program is neither client nor
server.

      - Microkernels are the way to go

As I understand it, there's a fair amount of reactionary sentiment
growing in the uKernel community, as things that were taken out of the
kernel are thrown back in for performance reasons.  It seems to me
that uKernels *are* the way to go, but we have to be careful what we
mean by that.  Specifically, uKernels (indeed, any software system)
must be designed in a highly modular fashion.  They should be
*conceived* of and *designed* as multiple plug-compatible units that
interoperate to provide the needed services.  It's probably more
evident in the case of an OS than in other systems that these units
should be largely autonomous and equipped with very flexible but
highly structured facilities for interacting with each other.  When
you actually produce a running OS from such a modular specification,
you (the compiler and/or run-time system, perhaps with your helpful
advice) may well choose to combine several units into a single
executing image, but this would be a *transparent* implementation
choice.  Inter-module calls could then be highly optimized to perform
as quickly as conventional procedure calls.  The high-level design
would allow a wide variety of implementation choices, from fully
physically distributed (with some resulting performance hits) to
physically monolithic.  Still, it's the same easily maintained, easily
reconfigured, nicely modularized system from a logical point of view.

Secure languages like Hermes can help in providing alternatives like
this, since in a secure language one need not be wary of combining
modules in this fashion.  Each module is known to be incapable of
causing another module to crash, whether or not they're separated by
address-space barriers and the like.

      - UNIX can be successfully run as an application program

Not all that controversial (except from a performance point of view,
perhaps), but it's certainly not something I dream of achieving.  I
hope the distributed systems we'll be using down the road will be far
less strewn with inflexibilities and unnatural models than Unix.

      - RPC is a good idea to base your system on

A lot of people have argued with this point, implying that RPC implies
round-trip communication and operating system overhead.  But let's
think in terms of a separation between logical and physical execution.
If I want to access a service provided by some other process,
procedure call (which may or may not be remote, I don't care) is a
convenient model for me to use.  It's an easy model to design into a
language, and people have been pretty happy with it for
non-distributed programming for years.  But maybe my compiler is smart
enough to turn my void-returning procedure call into a one-way
message, or buffer multiple successive calls into one with provisions
for rolling back in the unlikely event that an intermediate call
fails.  It's still RPC from a logical point of view, but it's
physically implemented for higher performance than some say RPC
implies.  The point is, physical platforms should not be of concern to
the vast majority of programmers.  Transparent program transformations
need to be developed to map simple, coherent concepts into efficient
implementations on a variety of platforms.

      - Atomic group communication (broadcast) is highly useful

I can conceive of applications where atomic group communication is
part of logical behavior of my program.  But I think most of the push
for this facility comes from its use in providing replication for
either availability, performance, or fault-tolerance.  In these cases
I'd like the compiler and/or run-time (possibly at my urging) to
arrange for replication and perform the necessary transformations to
achieve atomic group communication among the replicas, if that's
required in order to preserve the semantics of my program.  In other
words, I don't want to know that my program is going to run on
machines that can crash and over networks that can fail.  I'd rather
program in an ideal world where those things don't happen, and have
the failures transparently masked by the system.  So as a mechanism
for providing fault tolerance or boosting performance transparently,
I'd agree with the above.

      - Caching at the file server is definitely worth doing

Caching all over the place is worth doing when the cost analysis says
it is.  But again this should be transparent to the programs that
benefit from it.

      - File server replication is an idea whose time has come

True, but not just for file servers.

      - Message passing is too primitive for application programmers to use

Do you mean opening sockets and doing selects on file descriptors and
that sort of thing?  Then yes, you are right.  But if there's language
support for communicating typed data reliably across logical channels
between processes, then it's not primitive and it's not difficult to
do.  And RPC can (and should, in my opinion) be logically viewed as a
pair of message communications.  Viewing it this way, and integrating
it into your language model as such, allows for the sort of flexible
but structured process interaction model that I argued above is not
offered by client/server.

      - Synchronous (blocking) communication is easier to use than asynchronous

I generally agree with this statement, but noting that in a synchronous
world it's probably easier to develop systems that are prone to
deadlock.

      - New languages are needed for writing distributed/parallel applications

New languages are needed that allow programs with complex interactions
to be developed, without regard for whether they will run distributed
or parallel or whatever.

      - Distributed shared memory in one form or another is a convenient model

I prefer to view physical shared memory as an extremely
high-performance medium for program interactions.  I don't like shared
memory as programming model, as it *forces* the programmer to write
excess code (locks, or whatever) to make sure concurrent accesses are
properly managed.  I much prefer a logical model that does not expose
sharing, and a compiler that can make use of shared memory to boost
performance when it's available.  I don't much like the idea of
distributed shared memory, because, biased as I am against shared
memory as a programming model, I don't see the point.  I've sometimes
thought that the message-passing people and the distributed
shared-memory people are both trying to do the same thing: make the
distributed programming model and the non-distributed programming
model look the same.  Distributed shared memory takes the model we've
been using for standalone processing and tries to extend it to the
distributed world, so programmers can continue to use that paradigm.
The message passing people, it seems to me, take a model that is
inherently better adapted to a distributed world, and say that this is
how programs interact, whether or not the underlying platform is
distributed.  In either case there's a single model, which is a
simplification from where we started (shared memory for local
interactions, explicit network communications for remote
interactions).  I believe that the message-passing view gives a far
preferable logical model, as it promotes modularity and all its
benefits, while shared memory messes with module boundaries and
therefore complicates the world.

   STILL HIGHLY CONTROVERSIAL
      - Client caching is a good idea in a system where there are many more
	nodes than users, and users do not have a "home" machine (e.g., hypercubes)

I'm not sure what you mean by "client caching."  But on basic
principles I'll disagree anyway!!! :-)

      - Atomic transactions are worth the overhead

If the semantics of the program require atomic transactions, then
they're indispensible.  If the program doesn't require them, then
they're not worth even the tiniest overhead.

      - Causal ordering for group communication is good enough

I think this is probably true.  The problem is that there are ways for
information about causal dependencies to propagate that are not going
to be tracked by the software (e.g. I can tell you that my program
just printed a message, thereby *causing* you to perform a certain
operation that causes your program to send a message; the two messages
are causally related, but the software doesn't know that).  The
totally ordered atomic broadcast of ISIS (abcast) will correctly
reflect the world despite such "side-band" communication, but it's at
a rather high cost.  I believe that a model like cbcast is probably
sufficient for developing logically consistent and robust software.
	
      - Threads should be managed by the kernel, not in user space

I think there's room for both.  The key is that the programmer should
not have to be concerned with it.
--
Andy Lowry, lowry@watson.ibm.com, (914) 784-7925
IBM Research, P.O. Box 704, Yorktown Heights, NY  10598