From: jml4@cus.cam.ac.uk (John Line)
Newsgroups: news.admin
Subject: SUMMARY: Re: Request for ideas - news spool dumping strategy
Date: 18 Aug 92 09:38:20 GMT
Organization: U of Cambridge, England

This is a long-overdue summary of the (two) email responses I got to a request
for advice on dumping my news spool to tape, posted on 22nd July. In summary,
my situation is: Sun 4/390 with 1GB in use (and typically around 90% used) for
the news spool plus control files. The spool partition had just self-
destructed for no good reason during a reboot (thankfully not repeated since,
even during several power failures), and reconstructing the control files was
painful, apart from the loss of articles as no dumps had been done at all
(negotiations were in progress...).

No local tape drives; routine dumping is to EXABYTE (EXB8200, low density) on
a couple of remote systems, and is done automatically overnight (no-one to
change tapes). Fitting in level 0 (full) dumps occasionally wouldn't be a
problem, but frequent incrementals would (I was assured) risk (over-)filling
the daily incremental (level 9) tapes, which would also be receiving level 9
dumps from 3 other systems. Especially with the imminent addition of several
2GB drives on the other systems... My news feed amounts to around 40MB/day,
and expiry is 28 days for "technical" groups, 14 days for alt etc., (roughly
speaking); thus a daily level 9 dump could get up to 250MB or so of news data
just before the next weekly (level 5) or monthly (level 0) dump.

My thanks to jerry@strobe.ATC.Olivetti.Com (Jerry Aguirre) and Frank Nusselder
<murphy@cs.kun.nl> for the comments they sent by mail, which I summarise below
together with my thoughts on the ideas:

(1) A dump need not contain everything since the previous level 0 (or
whatever) weekly/monthly dump, e.g. do level 1, 2, 3, 4, 5, 6, 7, 8, 9, on
successive days (implying a level 0 dump every 10 days), dumping only updates
since the previous day. The cycle could be "stretched" by dumping on alternate
days, or you could do level 1 dumps several times between level 0 dumps.

This sounds plausible, but recovering the spool could mean restoring from a
large number of individual dumps. As long as it was rarely needed, not a
problem?

(2) It would be possible to feed the output of dump to compress. The specific
suggestion was to write the compressed file to another partition which would
be dumped later.

I wouldn't be able to compress to another partition, and I'm not sure if
compressing direct to a tape also containing normal dump output would be a
good idea. I tried dump | compress | wc to check the compression achieved, and
for my news spool it saved 60% of the space, but roughly doubled the elapsed
time (without even touching a tape) compared to a normal tape dump of the same
spool contents.

(3) A dump of a changing file system (e.g. news spool) may not be much use.

My feeling is that it would be better than nothing if the only problem was a
few missing files (as long as the missing files weren't important control
files, of course).

(4) If everyone were to dump their news spool, that would be a lot of
duplicated effort and redundant EXABYTE tapes. If feed sites keep articles at
least as long, arrangements to allow recovery by having them re-send the lost
articles could be an alternative. By arranging to dump the control files
(advisable anyway!), it would even be possible to reinstate the messages under
their original article numbers, avoiding problems with users' .newsrc files
being out-of-step with the server.

My feeling on this point is "yes and no". My spool has local groups which
don't exist anywhere else, and and if I have dump tapes from last night when
the partition dies, I'm not dependent on anyone else (or their systems) to
help get it back into service. People who have to pay communications charges
for their news feeds might not think it very sensible, either, though that's
not a problem for me - hogging the bandwidth could be.

If you arranged to have the feed site(s) build batches to send their entire
spool, what's the betting the news expert at the feed site will be on holiday
when you need help? Requesting individual articles via NNTP would be slow, and
in my case is not allowed by the feed sites. Also, judging by how long it took
(8 hours or so) to process 5 days-worth of old news batches that just happened
to be available when I was recovering after my spool died, how much elapsed
time and network bandwidth would it take to recover 28 days-worth of articles
that way, even if neighbouring sites kept the articles that long? A
cooperative arrangement between feed sites could be very useful in the right
situation, but I don't think it helps in my case.

In conclusion, the ideas were very interesting (thanks for the suggestions!),
but I still haven't managed to agree a dumping strategy with the person in
charge of dumping on the systems that own the tape drives. For now, I've got a
nightly cron job saving the control files onto a partition which is dumped
daily, excluding the logs and history files which are large and volatile, and
am doing "manual" level 0 dumps of the whole spool partition every few days
(sigh!).

I am now wondering whether to move the local (University-internal) groups onto
another partition, which is dumped, since many of those articles may be of
greater long-term importance to the users than articles from outside
(departmental administrative information as well as technical discussion), and
couldn't be recovered by any other means. This assumes that C-news wouldn't be
confused by part of its spool being at the far end of a symlink...

                                John
-- 
John Line - Cambridge University Computing Service, Computer Laboratory,
            New Museums Site, Pembroke Street, Cambridge CB2 3QG, ENGLAND.
Internet: jml4@cus.cam.ac.uk  JANET: jml4@uk.ac.cam.cus  Phone: +44 223 334708