From: leichter@lrw.com (Jerry Leichter)
Newsgroups: comp.os.vms
Subject: re; How to make VMSMail run lots faster
Date: 8 Nov 92 14:44:22 GMT
Article-I.D.: uu3.9211081545.AA23234
Distribution: world
Organization: The Internet


	We are having severe VMS Mail performance problems.  The problems
	center around users with lots (thousands) of mail messages stored as
	external files in the VMS directory where they receive mail.  Once the
	size of this (or any) VMS directory  passes 128 blocks or so (many of
	ours are >> 1000 blocks), performance for adding new files begins a
	rapid decline.  Other users sending mail to these accounts will
	encounter seemingly random long delays waiting for the "MAIL>" prompt
	after sending.

MAIL-11 is a synchronous protocol:  The sender won't see the prompt until
delivery is confirmed at the other end.  You could avoid this particular
delay by forcing all messages to go through a queueing mailer.  However, this
would add a great deal of overhead without really solving any significant
problem.

	Actually the delays occur exactly every seventh message delivered to
	the recipient account.  If the normal time to send a message to one of
	these accounts is 1-3 seconds, the seventh message will often take
	30-50 seconds and consume hundreds of DIO.  It is not an RMS problem,
	the size and organizational state of MAIL.MAI have almost no effect on
	these measurements. This behaviour is an normal consequence of the way
	large VMS directories work (confirmed by Colorado Springs).

The GENERAL property isn't unexpected:  If there is room for k directory
entries per block, on every k+1 new messages, on average, a new block has
to be allocated, and entries have to be shuffled around.

In the old days, the names of external mail files were chosen in increasing
alphabetical order.  This meant that new messages always got placed at the
end of the directory, minimizing entry shuffling.  However, it also caused
excessive directory growth given the usual pattern of mail deletion (most, but
not all, new messages are immediately deleted, leaving large unfillable holes
in the directory structure).  These days, external mail file names are more
randomly created, which fixes one problem but creates another:  Now, growth
of the mail directory file is INTERNAL, so when a block files, on average
half the blocks have to be moved "up" to make space.

The only thing that surprises me is that you see k=6.  The directory entry
for a MAIL$xxx file is 38 bytes long, plus 2 bytes for the length, allowing 12
in a block.  Ah, I see, at steady state, blocks range in "fullness" from half
full to entirely full, so if you add an entry to a block at random, about one
in 7 times it will overflow.  VMS will then allocate a new block and divide
the entries evenly between the old and new blocks, maintaining the statistics.

I'm still surprised that you see "every 7th message" PREDICTABLY - it should
be "on average".

	Possible solutions revolve around reducing the size of the VMS
	directory where the messages are stored:

	1)  Tell users they have to make their VMS directory holding mail
	    smaller (via archiving, deletion, etc.) (see also 4, 5, and 6)

	    Disadvantages:  
	        inconvenient for users

	        requires user retraining

I'm afraid this is the only real alternative.  Every program makes some
assumptions about expected file sizes and such.  The XQP makes assumptions
about "reasonable" directory sizes, and MAIL makes assumptions about the
size of "reasonable" mail files.  Unfortunately, MAIL's excellent performance
for even quite large mail files leads people to push it way beyond the design
limits of the overall system.  (As always, the more people have, the more they
want.  Someone with a thousand-block mail directory - assuming it has been
cleaned up properly, so doesn't contain tons of deleted, unusable space -
has probably got AT LEAST 10,000 mail messages saved away.  Try accessing
a mail file anything like that large with any other widely-used mail system.)


	2)  Patch MAIL to use shorter external file names and thus smaller
	    VMS directories to keep track of them.

	    Disadvantages:
	        limited gain for effort involved

	        we don't have source listing

	        a maintenance problem as new releases come out

I've been considering making such a patch myself:  It's a pain in the neck
to keep enough contiguous disk space around for mail directories!  A simple
job can almost certainly be done without the listings:  A quick look at the
mail shareable image shows to occurences of what is clearly the $FAO format
string for producing mail external file names.  If you know for certain that
you will always put all your mail in a subdirectory, so don't care what the
file names look like, you could reduce MAIL$xxxx.MAI to just xxxx.  This would
save 9 bytes, reducing each record in the 40 bytes (total) to 32; the number
of entries per block would go up from 12 to 16.  Not a huge gain, but probably
it would help some.

I quite agree that maintenance would be a problem.

(8 other solutions omitted; I agree, none of them is very realistic.)

I can outline one solution that's just slightly better than completely
separate mail files:  I have [.MAIL] and [.OLDMAIL] directories.  [.MAIL]
is my primary mail subdirectory, and contains mail MAIL.MAI file.  [.OLDMAIL]
contains MAIL$xxxx.MAI files that I've moved there when they got too old for
me to need continuing access.  I move these files by hand; the base entries
for them remain in MAIL.MAI.

Also in [.OLDMAIL] is a directory entry named OLDMAIL.MAI; it is an alias for
[.MAIL]MAIL.MAI.  A logical name, OLDMAIL, is defined as:

	SYS$LOGIN_DEVICE:[.OLDMAIL]OLDMAIL.MAI

All my new mail is directly accessible.  The envelope information for my
old mail is also directly accessible.  However, if I try to look at a message
whose MAIL$xxxx file has migrated to [.OLDMAIL], MAIL reports that the file
can't be found.  No problem:  SET FILE OLDMAIL, and the message can now be
read as exactly the same message number, in exactly the same folder.  (Of
course, now the NEW messages can't be found.)

This is a hack, and is not the way MAIL was intended to be used.  It works for
all except one case:  You must make sure, when you do a PURGE, that (a) all
the deleted messages about to be purged have external files in a single
directory; (b) you have SET FILE to MAIL/OLDMAIL in the appropriate directory.
Otherwise, PURGE will get rid of the only pointer to the message, but won't
be able to get rid of the message itself.  I have a slightly modified version
of Harvey Brydon's MAILCOUNT utility that can scan multiple mail directories;
it can clean up garbage left after doing an incorrect purge.

Also, of course, you have to updated the alias entry every time you do a
COMPRESS.  I do a COMPRESS from a command file on a regular basis, and it
sees to it that the alias is updated.

Let me suggest one other possible (kind of) fix:  It's absolutely certain that
most of these thousands of messages that people have stored away have not been
looked at in months.  Most will NEVER be looked at again.  Get a disk
archiving utility and move all unaccessed files - mail files included - off
the disk on a regular basis.  Bring them back only if people ask for them.

Good luck!
							-- Jerry