From: artg (Arthur P. Goldberg)
Newsgroups: comp.os.research
Subject: followup to Andy's second challenge
Date: 10 Apr 92 23:13:34 GMT
Originator: osr@ftp


In article <32507@darkstar.ucsc.edu> ast@cs.vu.nl (Andy Tanenbaum) writes:
   ... I hereby challenge all the people who have commented on my list
   to do better.  List what YOU consider the five most important
   things we have learned about distributed operating systems in the
   past 20 years.

This is what I think we should learn by looking at the distributed systems
people actually use.

1. Network transparency is a BIG win.
A distributed operating system should always try to make access to
remote services look like access to local services.  Look at the
distributed systems (here I disagree with Andy's definitions) that
have become most successful and widely used in the last decade:
network file systems, X, internet domain addressing, Novell netware, Notes,
yp, Athena, etc.  All of these provide an interface at which the same command
works remotely or locally.  (In NFS, access a file independent
of its location; in X start a client anywhere as long as the
DISPLAY parameter is set; etc.)

2. Fault tolerance is essential.
People have expected that distributed systems would be able to
provide fault-tolerance via redundancy.  I find that the opposite
is true.  We get less reliable service because we depend on
more systems being up, and because we can keep working in
partially failed environments.  A corollary is that one of the
disadvantages of having your own workstation is that a system
problem that afflicts only you gets less attention than one
which would have afflicted the multiple users of a timeshared machine.

Consider these examples.  Printing - it used to be that if my
machine was up and the printer worked I could print.  Now, my
machine, the network, another machine and a printer have to work.
File service - disks themselves are more reliable and much much bigger,
but file service isn't better.  Again, file service often requires
two machines and a network.  Debugging - system consultants have more
trouble solving problems because they deal with a more complex
environment, especially intermittent network problems,
buggy distributed service algorithms and machine
and operating system heterogeneity.

I'm reminded of Lamport's quote: "A distributed system is one in
which I cannot get something done because a machine I've never
heard of is down"
...  NFS server Pluto unavailable, still trying

3. Remote performance must be comparable to local performance
(within the user's ability to notice the difference).
Otherwise people will insist on bringing whatever data they're using
to their local machine to avoid the performance cost of being distributed.
For example, people do anything to get high speed modems for home.  Very
few people were willing do to interactive editing over the
old 1 sec round trip delay of the internet, but nobody minds
rsh'ing around the LAN, even if it means being routed through
several gateways.

I only worry about latency, or bits in the pipe, in terms their
influence on this issue.  If I'm reading hi-res
video over the net and each frame access is an rpc with a latency
of 15 ms, then if the video has 100 frames/s then I'm screwed.
But if a video spigot across the network is sending the frames at the
right rate then they can look just as good on my screen as if
I were reading them locally.

4. Any successful design must SCALE by an order of magnitude.
One experimental distributed system I worked on had a fatal lack
of scaling flaw.  It started out with 4 machines used
by researchers, performing well and making people happy.  Then
it tried to serve the entire department of 20 machines.  The
global state algorithms didn't scale well, especially those that
determined which machines were up.  And, to make matters worse,
all file system access halted during a membership calculation,
including stdio.  So whenever one machine changed state, everyone's
terminal locked up for many seconds.

Needless to say, the system did not stay in use for long.

5.  KISS (Keep is Simple, Stupid.)
The distributed systems that I've liked have had a simple model
at an appropriate level in the architecture.  We can argue about the
elegence, efficiency and cleverness of the algorithms implementing
the widely used systems I mentioned above, but all of them at least
have a simple model SOMEWHERE in the architecture.  For example,
X has an increadibly complex set of library calls (protocol
messages) but it has the simple model that a client just about
anywhere (given enough bandwidth), running on almost anything, can connect
to a server.

Ob plug: Our work, the languages for distributed computing, Hermes and
Concert/C, and the transparent fault-tolerance algorithm
Optimistic Recovery seek to address some of these issues.


(Note that I talk about distributed systems that communicate data
and coordinate people's activities, not parallel systems that
speedup solution of computationally demanding problems.  The two
seem very different, to me.)

Arthur P. Goldberg
H1-A11, Box 218, IBM Research Center, Yorktown Heights, NY 10589
voice: 914 784-7608 FAX: 914 784-6307
email: artg@watson.ibm.com