From: wkt@rodos1.adfa.oz.au (Warren Toomey)
Newsgroups: comp.os.research
Subject: Re: A challenge
Date: 11 Apr 92 01:56:21 GMT
Organization: Australian Defence Force Academy, Canberra, Australia
Originator: osr@ftp

In article <32507@darkstar.ucsc.edu>, by ast@cs.vu.nl (Andy Tanenbaum):

	[ who asks us what have we learned in the last 20 yrs ]

My 2 cents worth, but first background: young programmer who has spent the
last 3 years helping design and build a distributed OS frm the ground up.
Young enough not to have played on Version 7 Unix.

> 1. Microkernels are the way to go.

Very much agreed. Putting as much hardware/dependent code in one layer
make the whole thing easier to port, and gives the OS proper an abstract
machine to support.

>       The performance loss is (guess) less than a factor
>       of two.  If you worry about factors of two, I humbly suggest you write 
>       all your code in assembler henceforth.

Well, if you `hand tune' your ukernel, then you can achieve good performance
on all platforms and still keep the same abstract machine for the real OS.

[ And a question from a naive OS weenie. Why does Mach call itself a
  ukernel when it's HUGE? It should really be a millikernel or something]

> 3. RPC is a good idea to base your system on.
>          SEND/RECEIVE primitives make I/O your central paradigm.

Hell, make RPC your central paradigm for everything. The OS I've been working
on (RHODOS BTW) has exactly _seven_ system calls: send,call,receive plus
comms endpoint manipulation ones. Everything else is done by server processes.
Makes the OS more modular and flexible, and easier to debug.

However, make your message passing FAST & EFFICIENT. Copying more than twice
is right out. BTW, with shared memory, message passing can be used for
token passing/semaphores while the bulk of the information is passed through
shared buffers.

> 2. Synchronous (blocking) communication is easier to use than asynchronous.

Sometimes avoiding a bottleneck by using async comms is more important. But
as Andy points out, it isn't as clean, so the reasons have to be good. We
provide both types in RHODOS, at least for flexibility.

> 4. Atomic group communication (broadcast or multicast) is highly useful.

The provision of several types of multicast is even better, because each
has a different purpose:

	0 or more receive - non-blocking, no guarantee anyone will receive
		 	    it. Useful for transport protocols that want to
			    avoid state & can afford to retransmit a bit more.

	At least 1 receive - blocking. Useful where there may be N servers,
			     and a client only needs to get one answer back.

	All receive - as per the reasons Andy stated.

> 5. Distributed shared memory in one form or another is a convenient model.

I'll agree on the shared bit. Synchronising memory updates & access over
a network is a real headache.

In RHODOS we decided that threads were just too troublesome to do in the
kernel. Let the user process do the work itself, by using a library of
thread routines that form part of the user process.

Instead, we use a MEDIUMWEIGHT process model, where processes have their
own stack spaces but share a common data space read/write. This means that
we do normal context scheduling on all processes, but they can access each
other's memory - a flag in our fork() equivalent makes the parent/child a
mediumweight process pair.


I'll add another lesson to the list: file mapping. This is great! It gets
rid of multiple data copying (disk->buffer->user buffer), and means the
whole of physical memory can be used as a disk cache. Moreover, if you
use the paradigm of file-memory object equivalence, you can use the whole
of the disk for paging, instead of just one partition. And the migration
of processes and their memory can be done through a common file server,
allowing the moving of the dirty pages only.

	I expect flames from the elders. Go ahead, educate me :-)


    Warren Toomey VK1XWT, on the path to insanity
    No longer in the bowels of ADFA Comp Science.
 `I wants virtual memory! I wants it now, precious.'