From: leichter@lrw.com (Jerry Leichter)
Newsgroups: comp.os.vms
Subject: re: Memory quota problem
Date: 28 Aug 92 03:26:26 GMT
Article-I.D.: uu3.9208280325.AA19846
Distribution: world
Organization: The Internet


	We have a slight problem here.  We have an application (ANU-NEWS)
	which seems to be a real memory hog.  I have seen it take up to 6000
	pages of memory to run.  Our students are limited to 2048 pages of
	memory.  As soon as that is full, the process begins taking up large
	amounts of CPU time just to do the page faulting.  According to
	MONITOR the page fault rate tops 10,000.  What we would like to do is
	set up this one process to allow for a larger memory quota, but keep
	other programs running within the 2048 limit.  I thought there should
	be a priv for this, but I did not find one.  Does anyone have any
	suggestions?  The source of the program can be changed if neccessary,
	but that would not be preferable.

Yes, indeed, ANU NEWS is a big memory hog, AND it wanders all over memory
while running.

There is, unfortunately, no direct way to change the working set for one
image.  The best you can do strictly from VMS is to increase the WSEXTENT
you grant the students.  That way, when there are free pages to be had, they
will be able to use them; but when the system is tight on memory, they won't
hurt others, at least not as much as if they had a large WSQUOTA.

Beyond that, you need to look at ANU NEWS itself.  The first thing you should
do is get V6.1beta.  It has a couple of improvements that, while they don't
affect the AMOUNT of virtual memory NEWS uses, will cut down on the amount of
thrashing it does.  (The big problem is at startup.  The algorithms in V6.0
use O(N^2) time and repeatedly hit o(N^2) memory, where N is the number of
newgroups.  Beyond a certain point, performance just disintegrates.)  V6.1beta
is quite stable, but starts up much faster with many fewer pagefaults.

Whichever version you go with, try to limit the number of newsgroups you
support.  I know, everyone wants every newsgroup available.  One site I know
of used ANU NEWS (V6.0) but had to remove it when the number of supported
newsgroups on their NNTP server double to some 3200; startup was just impos-
sible.  There is a "FAST LOAD" option that was hacked into V6.0, but didn't
work reliably; for those people it worked for, it solved the problem (though
it does have some limitations).  "FAST LOAD" is officially in V6.1, and I'm
told it has been fixed.

If you are willing to do some coding, the following are two suggestions that
could dramatically reduce memory usage.  I'm sure you'll find many people
very thankful if you get them working:

	a)  There is an array, I think the g array (I don't have access to
		the sources right now) that contains information describing
		each newsgroup.  In each element of the array, there is room
		to store a full NNTP server name for the group.  If I remember
		right, 100 bytes are set aside for the name.

		In fact, I would be astounded if any site is using more than
		a few different NNTP servers.  It would make much more sense
		to store the server names separately, and just store a pointer
		in the array.  This should be a very simple change.  If you
		wanted to be really aggressive about it, you could store the
		server names in an array and use a short, or even a char, as
		an index into it.

	b)  Similarly, each element of the array contains the full text of
		the newsgroup name in a long, fixed-length field.  This means
		that a string like "comp." (with the period) is stored tens or
		hundreds of times.  It would make more sense to parse off,
		say, the first two or three levels of newsgroup name and
		replace them with a byte index into a series of arrays.  E.g.,
		comp.computers would be stored as two bytes, the first into an
		array of top-level names, the second into an array of
		second-level names under comp.  A byte value of 255 would
		serve as an escape in the unlikely case that more than 255
		different names were needed at one level.

		This change is likely to be more involved, and to have a
		smaller payoff.  To make the most of it, the actual field
		used to store the name should be of varying length.

							-- Jerry
Newsgroups: comp.os.vms