From: mark@infocomm.com
Newsgroups: news.software.anu-news
Subject: Re: Spreading News over multiple platters
Date: 16 Dec 92 11:56:18 PST
Distribution: usa
Organization: INFO COMM - Computer Consulting, Redwood City, Ca

In article <1992Dec15.083654.45656@kuhub.cc.ukans.edu>, sloane@kuhub.cc.ukans.edu (Bob Sloane) writes:
> In article <1992Dec14.164132.686@totaltec.com>, mark@totaltec.com writes:
>> Is there anyway to spread the actual news files over more than one disk?
> 
> Yes. I do:
> 
> $ DEFINE/SYSTEM/EXEC              NEWS_ROOT        DUA4:[NEWSMGR.DATA]
> $ DEFINE/SYSTEM/EXEC/TRANS=(conc) NEWS_DEVICE      DUA4:[NEWSMGR.DATA.]
> $ DEFINE/SYSTEM/EXEC              NEWS_MANAGER     DUA6:[NEWSMGR.MANAGER]
> $ DEFINE/SYSTEM/EXEC/TRANS=(conc) NEWS_MANAGER_DEV DUA6:[NEWSMGR.MANAGER.]
> 
> I suspect that you could put all four on different devices, although I
> haven't actually tried it.

There is MUCH to be gained by separating NEWS_ROOT and NEWS_DEVICE on 
different spindles.  NEWS_ROOT is where the large index files (NEWS.ITEMS and
HISTORY.V60) live.  NEWS_DEVICE is where the individual .ITMs for each message
are stored in the appropriate directory NEWS_DEVICE:[news-group-name]

Assumptions in various places assume that NEWS_MANAGER and NEWS_MANAGER_DEV 
are the same place only one being a rooted logical for the other, so these
should not be separated.

With any version of NEWS after V6.1Alpha4, no particular performance difference
can be realized by locating NEWS_MANAGER on a separate device from either
NEWS_ROOT or NEWS_DEVICE.  In fact, the transient nature of files in
NEWS_MANAGER_DEV, along with the potential large dynamic range corresponds well
to the pattern of use that the files in NEWS_DEVICE.  Due to this correlation,
the NEWS management procedures provided with DECUS uucp actually require that
they all be on the same device.

Now, the original question asked about the "actual news files" which I'm
guessing are all the various .ITM files for each posting.  

I guess that NEWS_DEVICE could be on a volume set, but volume sets are not
known for any performance gains due to I/O load balancing. 

From my experience, the best performance gains can be realized by:

	1) separating NEWS_ROOT from the other three (NEWS_DEVICE,
		NEWS_MANAGER, and NEWS_MANAGER_DEV which should all be on the
		same disk)
	2) being "generous" with global buffers for NEWS.ITEMS, NEWS.GROUPS
	   and possibly HISTORY.V60

Given the above "optimal" setup, any slowness with news file accessing is now
limited by performance limitations of ODS-2 directory accessing for large
directory files.  The following factors contribute to this problem: 1) usual
newsgroup traffic levels combined with standard expiration periods end up
with newsgroup directory files that typically get larger than Files-11 will
manage in it's directory caches. 2) the fact that newsitems file names are
simply monotonically increasing, and expiration is always deleting files from
the beginning of the directory list (the good news here is that at least we
don't end up with sparsely populated directory files).  These factors are the
primary cause of real slow newsskim processing.  During news ADD processing 
the I/O load to update the file headers and the newsgroup directory is
usually several times greater than the actual load to write the article data. 
This is primarily due to the fact that the XQP does I/O to ODS-2 critical
data structures (Directories and File headers) is always done with DATACHECK.

--
Mark Pizzolato - INFO COMM Computer Consulting, Redwood City, Ca
PHONE:	(415)369-9366	UUCP:  decwrl!infopiz!mark or uunet!lupine!infopiz!mark
DOMAIN:	mark@infocomm.com