From: sl@wimsey.bc.ca (Stuart Lynne)
Subject: biz.sco.binaries - bumpy startup
Organization: BC News and Mail
Date: Tue, 6 Oct 1992 07:35:46 GMT

Well I finally decided to turn the floodgates loose on Saturday night.
Halfway through the posting of the first set my news *and* uucp partitions
suffered a meltdown. 

It seems a test version of nntp I installed on Thursday decided to start
putting each incoming article into a separate file instead of a batch file.
And for other reasons we ran a bit short of space on the news partition on
Saturday afternoon so Cnews stopped unbatching. Unfortunately nntp didn't
stop accepting new articles. And because they where no longer being
unbatched we started getting copies from 3-5 places depending on the
particular news group etc. By around midnight - co-incidentally at the same
time I tried to do my posting - we ran out of room. By this time there where
somewhere around 20,000+ files in in.coming.

To make matters worse someone else on my system spent most of Saturday sending
himself copies - lots of copies - lots and *LOTS* of copies of the same mail
message. My best guess is about 10,000 copies. This gave him about 30,000
files in his uucp spooldir by late Saturday evening.

I don't know how many of you out there are aware of the performance issues
involved with dealing with directories with >10,000 files in them. Lets just
say that any process that tries to munge through such a directory doesn't
surface - for a long time.

Also around midnight my uucp cleanup started to run. Of course the first
thing it did was to run into the fullup uucp spooldir and start beating it's
way through. Of course it was just the last of many. Updatedb (from the gnu
locate command) was also stuck in there somewhere. Also there where
something like 10 newsrun processes stuck in in.coming. And the person that
had mailed himself all those messages decided it would be a good thing to
delete them. So he wrote a script to use uustat -k to kill each and every
one. Commendable, but it provided yet another process to delve into never
never directory land.

The interesting thing in all this is that even with the load average through
the roof (+40) performance wasn't all that bad. Quite passable in fact.
Which is why I didn't notice anything amiss until the posting software
started complaining and out of space messages started appearing on the
console. Then I turned on the X Terminal and just about died when I saw the
load average display. (I had been upstairs most of the evening and had just
come down and started the posting up first thing from the console.)

It was fun. Did you know it can take rm -r 3-4 hours to delete a directory
with 30,000 files in it? Did you know that a slightly overloaded 486 can
work it's way through about 3000-4000 individual news articles per hour?
Lot's and lot's of fun.

The end result of all this was that the initial postings where to say the
least a bit munged. I actually didn't think they got out. I did check at one
point. But it now appears that a couple did. So I've gone ahead and reposted
the lot. The news-id's are the same as the originals. So only the ones that
didn't get batched the other night should get re-sent. I believe they might
be a bit out of order.

One hint for unpacking. If you read news with trn check out the :e command
and use it at the thread level. Something like:

	:e /tmp

Will extract all the articles in the selected thread and place the results
in /tmp.  

I've tried to make things as automated as possible. But there was some 
annoyment from the beta testers about automatically invoking custom(adm)
from the shar file so I took left that out :-)


-- 
Stuart Lynne <sl@wimsey.bc.ca> ....................... UNIX Facsimile Software
Wimsey Information Technologies ................... moderator biz.sco.binaries
uucp login:nuucp passwd:nuucp .................. ftp.wimsey.bc.ca:~ftp/ls-lR.Z
PD Software for SCO UNIX .................. ftp.wimsey.bc.ca:~ftp/pub/wimseypd