From: billd@cray.com (Bill Davidson)
Subject: A way to fix a corrupt Cnews history file.
Organization: Cray Research Superservers Inc., San Diego CA, USA
Date: Sat, 5 Sep 1992 05:29:27 GMT

My history file has developed a few different problems over the last
several months which Cnews bitches at me about on a regular basis.  Not
wanting to throw away my expired article history, I devised a series of
commands to extract what are likely to be the non-corrupt lines and
rebuild the database.  It seemed that others might like something like
this and I haven't heard of one so I decided to cobble it into a
script.  This is really just a first cut.  I'm sure someone will want
to re-write it in perl ;-).

This is an inherently dangerous script to run.  If you are not careful,
you can mess your news system up fairly badly.  You'll only want to run
this in an emergency.  Before you run this, I would recomend turning
off newsrun and nntpd and then running this inside a locknews shell.
Also, make sure that you have enough temporary space to hold a couple
of copies of your history file.  There are places in the script to
redirect where the temporary space is used.  I should also warn you that
it assumes that you use the dbz that Henry did for Cnews.

The usual disclaimers apply.  Good luck.

#!/bin/sh
. /usr/lib/news/bin/config
PATH="$NEWSPATH:$NEWSBIN" export PATH
cd $NEWSCTL
#
# The way this is setup, /tmp needs to have approximately 2 times the
# size of the history file worth of free space.  If this is a problem you
# may want to change the TMPDIR's below.  If your sort(1) doesn't support
# the -T option, you may want to set TMPDIR1 and TMPDIR2 to a null string.
# TMPDIR2 gets used at the same time as both TMPDIR1 and TMPDIR3 but
# TMPDIR1 and TMPDIR3 do not get used at the same time.  Each represent
# the use of one full history file worth of space.
#
TMPDIR1="-T /tmp"
TMPDIR2="-T /tmp"
TMPDIR3=/tmp
TMPHIST=$TMPDIR3/history.$$
#
# get something reasonably sane for the history file
#
#	1. egrep - force a <> bracketed message id with an '@' in it
#		with a tab followed by a date and '~'.
#	2. sort - helps us find duplicate message-id's
#	3. awk - sends non-dups to standard out, dups to a special file.
#		Too bad that uniq isn't smart enough to do this.
#	4. sort - sort new history by date.
#
egrep '^<.+@.+>	[0-9]+~|^<.+@>	[0-9]+~' history | \
	sort $TMPDIR1 | \
	awk '{ \
		msgid = $1; \
		if ( omsgid != msgid ){ \
			print $0 \
		}else{ \
			print $0 > "/tmp/hist.bad.dups" \
		} \
		omsgid = msgid \
	}' | \
	sort -n +1 $TMPDIR2 > $TMPHIST
#
# Now that we have a relatively good history file, lets try to move it
# into our $NEWSCTL.
#
rm -f history.o
sz=`/bin/ls -l $TMPHIST | awk '{print $4}'`
mv $TMPHIST history.n
nsz=`/bin/ls -l history.n | awk '{print $4}'`
# Check to make sure that we actually stored the new history file
if [ $sz = $nsz ]; then
	#
	# Ok, we moved it in successfully.  Let's replace the old database.
	#
	mv -f history history.o
	mv -f history.n history
	rm -f history.dir history.pag
	#
	# Rebuild the dbz portion from scratch (the old one may have bad
	# records in it).  Count the history lines to make dbz faster
	# with -p option.  For significant sized history files, this
	# saves orders of magnatude in time.
	#
	rec=`wc -l history`
	dbz -p $rec history
	#
	# Clean up the files from dups we deleted from the history file.
	#
	if [ -f /tmp/hist.bad.dups ]; then
		cd $NEWSARTS
		awk '{for(i=3;i<=NF;i++)print $i}' /tmp/hist.bad.dups | \
			tr . / | xargs -t rm -f
		rm -f /tmp/hist.bad.dups
	fi
else
	echo "cleanhist.sh failed."
	rm -f history.n /tmp/hist.bad.dups
fi