From: woods@robohack.UUCP (Greg A. Woods)
Subject: Re: in.coming on a different partition?
Organization: Elegant Communications Inc.
Date: Tue, 25 Aug 92 05:55:33 GMT

In article <1992Aug23.135448.4949@DIALix.oz.au> peterw@DIALix.oz.au (Peter Wemm) writes:
> I had an argument with somebody a few days ago....  The machine that
> I am posting from has disk space problems.  If a large burst of news
> comes in, /usr/spool/news fills up and batches are lost.  One of the
> people here has rigged up a complex system to catch batches that
> wont fit and put them on another partition.  It "works", but
> has a habit of leaving lots of uncompressed batches in in.coming
> which can't be uncompressed due to no space.

Ah, the dreaded catch-22 spacefor problem!

I've just gone through cleaning 20 Mb of incoming batches from a 50 Mb
spool partition!  (This is the second time through in the past few
years, and this time I was able to watch an analyze it more carefully.)

The basic solution, with C News anyway, is to be sure your spacefor is
properly configured to allow un-batching when there's less space than
for spooling.  I'm using a custom C-code spacefor with a test option:

	$ spacefor 1 CHECK
	incoming:    2000, /usr/spool/news
	articles:    1500, /usr/spool/news
	control:     5000, /local/lib/news
	outbound:    2000, /usr/spool/uucp
	archive:      100, /usr/spool/news

where /usr, /usr/spool, /usr/spool/news, and /usr/local are all
separate partitions (and /, /tmp, /home, & /usr/src too!).  The
difference between incoming and articles should be larger than the
largest average *uncompressed* batch.

Here's some arbitrary figures that show approximately how I've
parcelled my disks out:

/		total:    21060 blocks    2624 i-nodes
/tmp		total:    40800 blocks    5088 i-nodes
/usr		total:    95850 blocks   18000 i-nodes
/usr/spool	total:    48600 blocks    6064 i-nodes
/usr/spool/news	total:   100764 blocks   44064 i-nodes [15000 inodes too many!]
/local		total:   140130 blocks   17504 i-nodes
/src		total:   159800 blocks   19968 i-nodes
/home		total:    75820 blocks    9472 i-nodes

The other essential features are turning on rnews.immed and
rnews.stall, then modifying newswatch to run every hour, and call
'expire -r' when there's not enough space for an average incoming
batch of articles (`spacefor 100000 articles` -le 0).

In actual fact I keep several explist files in $NEWSCTL/explist.d, and
newswatch calls each in sequence until there's enough space.

Newsrun must also remove temporary files and already un-batched
batches from "incoming" as it works (as I believe the current C News
does -- I've modified my version to do so anyway).

If you're daring, and your news neighbours are all non-BSD sites, you
might also want to use the taylor-uucp uuxqt and add a grade option
such that you can run a periodic uuxqt that won't touch rnews jobs (in
order to expedite rmail processing).  [Actually, it seems tuucp's
uuxqt may already suffice with the '-c' option.]

All of these steps are essential to run C News in a robust manner on a
busy system that's tight on disk space.  I've also had to fix
sendbatches so that it won't try running batchsplit if there's not
enough space to split the togo file(s).

> Currently, news processing is a large drain on the system
> performance.  When the news spool fills up, it really begins to
> crawl.  Would moving in.coming help with our overflow problems?
> Would it improve or degrade our already streched system
> performance?

Putting $NEWSCTL, $NEWSARTS/in.coming & $NEWSARTS on different
spindles, *iff* your disk driver gets better performance, will
certainly help, as will having sufficient memory to avoid paging while
all possible news processes are running.  Otherwise splitting things
further than I have is probably not much help.

Get a copy of mawk-1.1 and install it in $NEWSBIN/awk.  It'll make
lots of scripts (the batcher especially) run faster.  If you're on a
SysV machine, replace instances of egrep with grep where possible.
The AT&T egrep has a great deal of overhead in startup, even with
simple patterns.

There are also probably still a few places (such as in queuelen?) [in
the pre-performance release) where the shell's white-space handling
will work far faster than piping through 'awk "{print $2}"', etc.

The final fix, of course, is to ask your news feeds not to send
anything your way while your system is serving other needs (i.e.
business hours).  The simplest way to do this with C news is to add a
modified queuelen to the out.special/sitename directory that returns a
large number starting some time before your "prime-time" start and on
through the day.  Then nothing will even be queued, thus e-mail may
continue to flow un-impeded.
-- 
						Greg A. Woods

woods@robohack.UUCP woods@Elegant.COM  VE3TCP	UniForum Canada & ECI
+1 416 443-1734 [home] +1 416 595-5425 [work]	Toronto, Ontario; CANADA
Xref: sparky news.software.b:2822 news.software.nntp:1050