From: leichter@lrw.com (Jerry Leichter)
Newsgroups: comp.os.vms
Subject: Re: RE: Clustered Global Section
Date: 26 Aug 92 01:34:37 GMT
Distribution: world
Organization: The Internet


	[The original question in this stream was about using disk sections
	to create "shared memory" across nodes in a cluster.  I wrote that
	it wouldn't work.  Mr. Naecker agreed but added some more informa-
	tion about a situation in which it COULD be made to work.  The
	following are minor comments on HIS comments, and won't make sense
	if you haven't read the previous messages, which are too long to
	repeat.]

	The basic idea is this:  Set up a lock to coordinate access to the
	global section between cooperating processes. When an application
	updates data that it wants to make available to other processes (on
	any node), it simply uses the $UPDSEC system service to write the
	appropriate addresses to backing store on disk.  This synchs the disk
	file with memory, at the cost of one disk IO.  Next, the updater takes
	out a lock that uses blocking ASTs to notify the other (interested)
	processes that an update has occured.  The lock value block includes
	the address range of the interesting updates.

	[Now each node waiting for updates reads the disk image back again.]

	The really tricky part is that seemingly innocuous line "updates
	memory from the section file".  There is no system service to do this,
	for architectural reasons dating back to VMS V1.0.  (This direct from
	a long discussion one DECUS with Larry Kenah.)  The easiest way to do
	it is to (gulp) post a QIO from the correct addresses in the section
	file directly to the correct addresses in memory.

I see no reason for the (gulp) here.  A file is a file.  HOWEVER, there is
a different difficulty:  When you open a file for access as a disk section,
you have to specify the RMS UFO (User File Open) option.  This tells RMS to
assign the channel in user mode (otherwise, it will assign it in exec mode
and you won't be able to use it directly).  However, a file opened UFO cannot
be accessed further using RMS!  So you either have to do direct QIO's to the
disk driver (legal, since the channel is assigned in user mode), or open the
file on another channel WITHOUT UFO, then use RMS block mode.  I haven't
thought through all the implications of the latter approach - the last thing
you want happening is for RMS to give you a cached, obsolete copy of the
block!

An alternative, if most changes to the section are "small", is to pass the
actual change in the lock value block.  There's plenty of room to send over
an address, a length, and a new value.

							   Except for a small
	offset at the beginning of the section file (four bytes, as I recall)
	the addresses are a direct map.

No; access to a section file is through the virtual memory/paging system.
Pages on the disk must align exactly with blocks in the file.  You CAN
specify a starting block position within the file, but the starting position
is in blocks, not bytes.

				         When this QIO has completed, then
	memory on that node will now reflect the contents of the section file,
	which, as you recall, was recently updated to reflect the contents of
	memory on the updater node.

	There are some possible gotchas here with respect to privileges,
	depending on the nature of your section file, but nothing that can't
	be handled straight-forwardly.

The only privilege issues I can think of are the channel mode ones I mentioned
above.  On the other hand, getting good performance while keeping the syncro-
nization correct will be tricky:  You can't allow an update of a page to be
made locally while you are in the process of reading it in from the disk.
You can sequentialize everything by forcing all accessing processes on all
nodes to wait until all signal that they are done, but that will obviously
be quite slow.  A more sophisticated method is called for.

	As I said, I've done exactly the above on two occasions.  With good
	application design, you can in fact use section files and two disk IOs
	to keep memory on different nodes in synch.

It's worth pointing out that this idea has seen a great deal of work in the
academic literature under the same "distributed shared memory"; the seminal
work is Kai Li's Yale PhD dissertation.  Imagine that, rather than simply
creating a shared, distributed section, you actually share all of P0.  In
effect, create a bunch of threads out of many processes running on multiple
processors.  (This is a ROUGH approximation, and wouldn't work in quite this
way on VMS.)  Kai Li did that by modifying the pager for Apollo's domain.
This approach has some interesting implications; for example, you can move
a process by just telling some other processor to run it.  It will fault all
the code and data in as needed.  (You have to "pin down" things like channel
numbers so that you can find files when you need them.)

Clearly, some things will not work well in this kind of setup.  Surprisingly,
however, many things do - but you have to work hard at making the paging
system clever about how and when it moves pages.  For example, you will want
to replicate read-only pages, while propagating changes to writable pages.
All this is quite tricky, but it CAN be done.
							-- Jerry