From: leichter@lrw.com (Jerry Leichter)
Newsgroups: comp.os.vms
Subject: re: Re: $add_holder is slow...
Date: 31 Dec 92 20:23:41 GMT
Distribution: world
Organization: The Internet


	[M Darrin Chaney writes:]
	We've got about 40,000 users on our cluster, which consists of various
	newer VAXen (the smallest being a 6500, I think).  Anyway, we still
	keep a separate rightslist and uaf on each of the six nodes (I don't
	run these machines, so don't blame me).

	Recently, I set about rebuilding the rightslist for the cluster, in
	order to get each node in sync again, and make sure that everybody has
	only what they need.  We have 5 different rights, of which everybody
	has at least one.... 

	Anyway, we had a spare 8820 sitting around (VMS 5.5-2), so I'm using
	it.  It has 384MB of memory, and we set up a huge cache on it.  We
	also made the account that we are using have high quotas....

	Anyway, I copied the rest of the identifiers into the new rightslist,
	of which there were only a few thousand.  Then, I used the cluster UAF
	to create the username rights (using add/ident/user=* in authorize).
	I made 5 files, each containing in "rightslist" format the UIC of each
	person who needed that right, and the right itself.  So, each line was
	like such:

	right,uic,attr

	attr is always 0.  The program loops through the file, reading each
	line and using $add_holder to add the holder....

	The amount of time it takes to add these increases rather
	dramatically.  The first 50 (of any set) take maybe 10 seconds.  After
	around 1000, it takes 10 or 15 minutes to do 50.  I figure it's
	because of RMS keeping track of all the duplicate keys.  I have
	everything except UGRAD added, with about 16,000 left.  At the current
	rate (50 every 20 minutes), it'll take another 4 days.  It will
	probably slow down even more, though, so the time will be longer.

	Does anyone know a way to speed this up?  Pretty much anything is
	possible with the machine that we are using.  Thanks in advance.

	[James Harvey responds:]
	Dramatic is indeed the word!  We had a very similar situation here
	about a year ago, but only with about 1/10th the users.  If you look
	at the organization of the rightlist file, the problem becomes clear.
	With thousands of users each holding one of a small set of "user-type"
	i.d.s, you end up with thousands of duplicate keys in an index slot,
	which are searched sequentially for an update.  We ended up storing
	the user-type information in a separate file indexed by username (you
	could probably store it in the "user info" field of the UAF now, which
	wasn't available to us at the time).  For all logins a privileged
	program is run that does whatever we want with this information (it
	defines an executive mode job logical here at our site, however, maybe
	you could use it to grant the appropriate identifier to the process at
	yours?).

Mr. Harvey's solution is a nice one, though it does mean that such services
as $FIND_HELD will not work as expected.

I'd suggest that Mr. Chaney's problem calls for stepping a bit outside the
pale of officially supported steps for a simple solution.  The format of the
RIGHTSLIST file is quite trivial.  Create a couple of sample entries covering
all the kinds of types you'll be adding.  Then use CONVERT to create a
sequential file from the resulting RIGHTSLIST.  Use the entries you have to
create all the rest of the entries you need.  Finally, CONVERT back to the
original format.  CONVERT builds the index structures intelligently; it
doesn't add the records one at a time.  It should run quite quickly.

(You may want/need to sort on the primary key before converting back - it's
been too long since I did this kind of thing for me to remember....)

							-- Jerry