From: davison@borland.com (Wayne Davison)
Subject: Re: relaynews vs the "threaded" newsreaders
Organization: Borland International
Date: Thu, 27 Aug 1992 01:25:53 GMT

geoff@world.std.com (Geoff Collyer) wrote:
> My earlier bold prediction, of feeding modified versions of mthreads and
> nnmaster with the headers of articles as they arrive, is unlikely to be
> realised.  After staring at mthreads (the harder case), it appears that
> the trn database is organised such that it cannot be efficiently updated
> as articles arrive

Exactly.  Trn's thread files were designed as a space-efficient snapshot
of the entire group organized via Reference lines.  Space efficiency was
the overriding design goal (the machine it was developed on had about 60M
of space dedicated to news :-).  So, I'd agree that feeding mthreads single
articles isn't a good idea (though I know of one fellow who has modified
C news to do exactly that).  What's needed is a new database format.

> Further staring has
> convinced me that in fact trn wants two distinct databases:  a
> ``presentation menu'' database [...] and the link data used
> to generate the trn `hunt the wumpus' diagrams

	Ieeeee.  You fell in a pit (known as "usenet").   :-)

Yes.  This conceptual separation is something I mentioned to Rich Salz
when we talked about his desire to integrate a news database into INN.
Whether it translates into a separation of files or not is up to you.
If it is in one file it must be easy to get in and update the parent
references as articles arrive.

It would be interesting to see the "relations database" kept on the
entire news-base so that trn could be enhanced to follow threads as
they wind their way through various groups.  However, a per-group
set of relations would be adequate to support existing readers.

Maybe you and Rich should get together and talk.  Last I heard he was
planning to return a set of headers from the group's articles including
special headers that would indicate their relationships.  Using this
concept, trn would ask for a dump of the group's data and it would link
up the parent/child/sibling references as it goes.  If the references
were given in terms of article numbers it would be a breeze to modify
trn to allow this.

> Does anyone understand what's *in* the link data in the trn database?

I know of at least one person who understands trn's database -- beyond
that I can't say.  Oh!  You wanted me to _explain_ it to you!  Ok.

Here's a simple summary of everything that is in the database.  If you
want a more detailed description of mthreads' database format, mail me.
I'm going to write up something for a VMS guy, so I could send you a copy.

The essential data that trn will need for each article includes:

	Article Number, Subject, author, date (in GMT unix seconds),
	parent pointer, thread-id, and flags.

The other data that is in the thread files that wouldn't need to be
supplied to trn:

	First child pointer, first sibling pointer, message id.

The thread-id is a unique id that indicates which thread an article
belongs to.  While an article's group can be determined by looking at
its lineage, it is essential to have an unchanging thread-id so that
kill files can act upon thread groups.  Mthreads puts this id in the
"root" -- a group node that contains a few items such as the subject
list required by the associated articles.

The parent reference is rather complicated, since it might point to
a valid article, or it might point to a missing article that has expired
or not arrived yet.  Mthreads uses "faked" article nodes for these
cases that are filled in with actual data when/if they arrive.

The flags that trn uses so far indicate if an article is the root article
or not and if an article has xref data available (it speeds junking).

The first- child/sibling pointers exist in the database because it is
doubly-linked.  These could be ommitted in the server database and trn
could double-link its data as it reads it in.

Mthreads keeps the message-id data around for each article for its
reference-line parsing, but trn doesn't use this data when news-reading.

If you want to keep this simple, you could provide something like the
following.  Assume trn has requested the thread data for n.s.readers
and it gets a list of headers back:

	Art: 1823 1432 missing1 missing2 1800
	Subject: trn
	Author: Wayne Davison
	Date: 713738170
	Xref: news.software.b:3987
	Art: 1824 1432 missing2 1800
	Subject: mthreads (was Re: trn)
	Author: Joe Smoe
	Date: 713734987

The "Art" line gives the article number (1823), the thread group id
(1432 -- the first article number we encountered for this thread),
and the parent list.  The first article's immediate parent and
grandparent are missing and must be specified by some sort of a
unique id so that trn's thread display can link things up correctly
(articles may have missing articles in common).  You might just want
to put the uninterpreted messages ids there, but they can get pretty
long.  A hash might be better.

> (I think I've got a block against `hunt the wumpus' diagrams.)

Oh well.  You can turn them off, you know.  I think the data provided
in the display is very useful, though.  If everyone had such a display
the number of duplicate responses to articles would drop appreciably.
-- 
Wayne Davison
davison@borland.com