From: dawson@willard.UUCP
Newsgroups: comp.bbs.waffle
Subject: HISTORY
Date: 29 Aug 92 17:13:26 GMT
Organization: Willard's House BBS, Atlanta, GA -- +1 (404) 664 8814

I've written a little program to do HISTORY database management... and
I've extended it to do expirations based on the Date: header.  My plans
are to further extend this program to handle control messages, e.g., to
do cancellations.

I might even go so far as to base a threaded news reader from the same
history database.

(Sorry if some of this looks familiar to a particular person (hi, Bill),
but I couldn't help but think that it would make a good intro to my stuff...)

Yes, my program is disk SPACE intensive.. as directly in opposition to the
ANTIDUPE stuff I hacked together, which was TIME intensive.  To give you an
idea of just how disk hogging it really is.. my current d:/news spool area is
at around 20.319 Megabytes... and my current history database looks like this:

HISTORY  DBF   1667995 08-29-92   9:51a
CONTROL  NTX     99328 08-29-92   9:51a
HISTORY  NTX   4542464 08-29-92   9:51a
MESSAGID NTX   4598784 08-29-92   9:51a

Unfortunately, I don't have recent timings to compare.. but ANTIDUPE would
just about NEVER finish searching my news spool for duplicates, whereas a
complete EXPIRE run of my HISTORY program now takes about 15 minutes.  (And,
you certainly don't want EXPIRE to be a part of RNEWS, do you?)  You won't
avoid having a history database file, but you might be able to avoid it
being as large as mine seems to grow.

No doubt the greatest weakness that Clipper and other X-base products has is
the lack of variable length fields/indexes.  Witness the size of my index
files...  some of my indexing could perhaps be simplified.. but not by much.
Certainly, all the disk space in use by the *.NTX files could be saved by
not performing SEEK operations, in which case there would be no reason for
the index files.  BUT, the time required to search the database for msg's to
expire (based on Date: headers, anyway) would grow quite a bit.. as LOCATE
operations are _nasty_!  (I suppose someone out there will be so short of
disk space that you'll want me to rewrite it do avoid using SEEKs and
index files... *sigh*.)

The biggest complication to using X-base, fixed-length field, database
engines to implement Usenet news software is that all the header fields are
variable length, and can even span multiple lines.  So, if you leave enough
space for long Message-ID's, et al., you wind up wasting a lot of space for
those messages that don't use it all.

[ Waffle handles this by ignoring it (just try to reply to an email whose
From: header has been line wrapped!).  Which makes life for us deep down and
far from a hub quite a pain in the butt. ]

[ I'm NOT about to use c-tree (Faircom) stuff, even if they do have variable
length messages, compressed indexes, etc.  Their stuff just outright sucks,
IMO.  It's buggy, undocumented, buggy, and undocumented.  I've had some very
poor experiences with it.  Maybe something else.. I've heard some good things
about some other packages, but I'll have to get some in for evaluation...
there's no use in sending good money after bad. ]

Perhaps something in between these two extremes (disk vs. time) would serve..
but I haven't felt too inclined to start messing about with the sources to
waf165, which I only recently received.

I don't feel a bit satisfied with the so-called source distribution as there
are just too many missing elements, i.e., rnews, rmail, uucico, uuxqt, etc.
This is all really rather annoying.  The LAST thing I (personally) need sources
to is WAFFLE.EXE.  Indeed, I could totally replace WAFFLE.EXE with HISTORY.EXE,
with the messages being stored in a MEMO field.  All I'd need to do to make
that happen is to wrap RNEWS/RMAIL/etc.  with a program to import/export stuff
from a TEMPORARY news spool, which I'd then erase.  Threaded news reading?
Simple.. just add an index on Subject:
[ 
    well, it'd really have to be somthing like
        INDEX ON STRIP_LEADING_RE's(subject) TO (cSubject_index)

    or some such, so that >'s and >>'s would show up later than the orig. msg.
]

Recreating the wheel is so much more FUN.. and that's what I'm doing this for
anyway, right!?  Like I'm gonna get rich from it!  Heh.

> 
> > It was all quite easy to write, and currently occupies some 845 lines of
> > Clipper code.  It will probably grow to just over 1000 lines to finish the
> > task of handling control messages.

I get 6 Usenet feeds.  Quite cluttered until I hacked ANTIDUPE together..
I just simply could not live with it's inefficiencies.  It was probably well
along it's way into eating years from my hard drive's life expectancy.  So,
I absolutely was forced into re-writing this HISTORY database thingy.

If there is any interest, I've placed a copy on hogbbs, and Bill Fenner
tells me he's put it into his files area.  Perhaps it will eventually wind
up on an FTP site, but I really don't care, as I can't ftp.  So, if you
want it, look for history.zip (or email me, and I'll see about emailing
an uuencoded version).

-- 
dawson@willard.UUCP (Willard Dawson)
Willard's House BBS, Atlanta, GA -- +1 (404) 664 8814