Newsgroups: vmsnet.mail.pmdf
Subject: RE: REPOST (for lack of any answer) performance problem with local
From: Ned Freed <ned@innosoft.com>
Date: 06 Oct 1992 12:15:03 -0700 (PDT)
Organization: The Internet
Return-Path: <epmdf@YMIR.CLAREMONT.EDU>
Resent-Date: 06 Oct 1992 12:15:03 -0700 (PDT)
Resent-From: epmdf@YMIR.CLAREMONT.EDU
Errors-To: epmdf@YMIR.CLAREMONT.EDU
Resent-Message-ID: <01GPMMDJKH5U8WWMTD@YMIR.CLAREMONT.EDU>
X-Vms-To: IN%"ECF_STBO%JHUVMS.BITNET@ymir.claremont.edu"
X-Vms-Cc: IPMDF
Mime-Version: 1.0
Content-Type: TEXT/PLAIN; CHARSET=US-ASCII
Content-Transfer-Encoding: 7BIT

>  It sounds like you've done most of the important things to speed
> things up. I would also recommend installing LIST.EXE /open/head/share because
> it gets run most every channel job. This code probably should have been made a
> routine called by the channel programs, but perhaps to retain compatibility
> with older channels, this wasn't done.

It _was_ done for every channel where it was possible to do so. Unfortunately,
due to bugs in VMS MAIL it isn't possible to do this for local delivery.

Simply put, there are roughly two dozen situations where VMS MAIL punts with a
fatal internal error. Once this happens there is no way to continue execution
in the same image. It is therefore essential to have some external way to
maintain context. The use of LIST.EXE plus the DCL glue around it all provides
this context. If we eliminate this context we lose the ability to recover
gracefully from these sorts of VMS MAIL failures, and the result would be that
one such error on one single file would stop up your entire local queue.

> Unfortunately, image activations and
> file creates/open/closes/read/deletes are a large part of the problem. You said
> you are running batch jobs on the fast cpu to do the local channel. If you're
> really doing standard batch jobs, I would recommend switching to process
> symbiont jobs here too. The batch jobs have to log in, map dcl and run sylogin
> and the system account's login.com every job, which is too rich for my blood.

The process symbiont is the right way to go.

> I
> have some problems with the process symbiont. It sits in a 6000 page working
> set for some reason.

You should have reported the problem. I suspect it is one we have seen before
and we have a fix for it. You should contact service@innosoft.com to pursue
this further.

> A couple of times since I've installed 4.1 the queue has
> stopped for reasons I can't determine. I find no dump file in sys$system...

This is probably a VMS problem of some sort. I've seen similar things happen
to all sorts of queues.

> I
> might go back to CTLSMB which is what I used before 4.1, I'll have to
> investigate further.

A number of changes were made in PMDF V4.1 that may cause this combination to
fail. We only support either batch queues or the process symbiont. In the past
although we didn't support CTLSMB or detached processes we went considerably
out of our way not to break such usage. This is no longer true.

>   Since you're not CPU starved, you may want to try to put images on a RAM
> disk. There is at least one publicly available one, and DEC has one too. Even
> when things are installed shared, open etc... there is some level of page reads
> to the disk. SET.exe and some others are the most active files on my system
> disk here. I don't know if it is copy on reference page faults or what, but I
> think VMS should be able to do better...

The long-awaited VBN cache hopefully will be part of VMS 6.0. This should
eliminate the need to do this sort of stuff.

>  If you use domain.dat (I put everything in pmdf_config_data) it might help
> a little to put some global buffers on the file.

This is a very reasonable thing to do. Similiarly, the QUEUE_CACHE.DAT file
can also benefit from such treatment.

> Another thing I've done to
> reduce the incredible amount of I/O is modify the task_server.com procedure.

I/O is PMDF's stock in trade. It is fine to want to reduce it some, but
always keep in mind that this is what PMDF does.

> Out of the box, the thing makes a log file every job! I was getting literally
> thousands of log files before a periodic job would come and delete them all
> (which is another expensive operation). This type of logging will probably
> be supported in the future. Also: I turned off verify. You still can see the
> error messages if there are any, and you can always turn it back on if there's
> a problem.

We're examining this area carefully for the next release.

>  The biggest traffic problem here, and there's no reason to think your
> site is any different, is BITNET mailing lists. Make sure your local listserv
> is not exploding mail to multiple recipients. It's faster to let PMDF expand
> locally than have it deal with all those individual messages. If you can
> encourage users to read lists on USENET rather than have mail deleivered to
> their account, you can cut down on traffic.

Agreed, although there are non-trivial performance concerns with news reading
too.

> All in all, I'd say PMDF 4.1 has
> much better worst case (millions of queued files) performance than 4.0. For
> regular non worst case it seems maybe a little better, or about the same. I
> guess the functionality enhancements balance out the performance improvements.

You have identified the design center for V4.1 quite accurately here. There
were only enough "inner-loop" optimizations to balance out the added
functionality. Most of the real work went into adding capabilities for systems
that operate under extremely heavy loads.

One unanticipated result has been that PMDF is now much more sensitive to
underlying system tuning and performance bottlenecks. For example, the
QUEUE_CACHE.DAT file is very sensitive to badly fragmented disks. It gives
higher performance than reading a bunch of separate directories does, but
a bunch of separate directories are much less sensitive to fragmentation.

In the future you can look for a bunch of additional "inner loop" optimizations
that hopefully will exceed the overhead caused by increased functionality.

We're also examining tuning issues with an eye towards providing much more
guidance in this area in the documentation. The central problem is that
such advice tends to be highly system-specific, and it is quite difficult
to come up with "generic" advice.

We're also looking into what sorts of performance monitoring we can do to
aid in the isolation of bottlenecks. Our problem here is that we cannot rely
on any single performance monitoring tool, so it is once again difficult to
come up with generally useful advice.

				Ned