From: rhys@meteor.brisnet.org.au (Rhys Weatherley)
Newsgroups: comp.bbs.waffle,comp.bbs.misc,alt.bbs,news.software.readers
Subject: Re: .QWK mail reader for Waffle
Date: Sun, 13 Dec 92 09:41:15 +1000
Reply-To: rhys@cs.uq.oz.au
Organization: meteor - Home of Helldiver and UW/PC
X-Newsreader: Helldiver 1.08 (Waffle 1.65)

In <1992Dec09031637nrc@bsbbs.columbus.oh.us> N. Richard Caldwell <nrc@bsbbs.columbus.oh.us> writes:
>I had heard that there was some talk of updating the QWK standard.  It
>seems to me that if you could designate a particular value for the message
>status flag to indicate that the message text is a USENET style (I forget
>the RFC number, sue me :) message including headers, it *might* not break
>current QWK mail readers.  That way, future QWK mail readers could handle
>news articles and mail more effectively.

You mean like this ... (I threw this together a month ago - it may see
eventual support in the Helldiver newsreader, and I have some Unix code
for generating the packets from an NNTP server)

Distribute and comment ...

Cheers,

Rhys.

------------------------------------------------------------------------
			Helldiver Packet Format Version 1.0

			Copyright (c) 1992 Rhys Weatherley

			        rhys@cs.uq.oz.au

				  November 1992

INTRODUCTION

For many years, the FidoNet community has been using QWK and other formats to
enable users to download their mail and conferences to be read while off-line.
This not only saves phone charges and prevents tying up BBS lines for long
periods of time; it also allows a user to use much more powerful tools on
their own machine to process the downloaded "packets" than what can be made
available in an on-line environment.

To date however, very little work has been done in the USENET and dial-in Unix
community to facilitate the same user operations.  Some attempts have been
made to use QWK, but due to QWK's limitations and unsuitability for the USENET
message formats, such efforts have not been very successful.

Within USENET, the trend seems to be either "dial-in to some other machine
and put up with it", or "set up your own USENET site".  The former keeps the
user at the mercy of out-dated user interfaces, and the latter requires far
more computing knowledge than the average computer user is expected to have,
effectively locking out large portions of the public from experiencing USENET.
The latter option can also give rise to security problems in the form of
forged USENET messages, which a more controlled dial-in system avoids.

The purpose of this document is to define a new packet format which is aware
of the conventions used in the USENET community, forming a middle ground
between dial-in user interfaces and full USENET connectivity.  It is not
limited to downloading USENET news however.  The same format could be used
to enable a Unix user to package up their Unix mailbox and download it for
later perusal.  The format is extensible to other kinds of news or conference
systems, so it is feasible, although not yet defined, that QWK or FidoNet
messages could be accomodated within the same packet as USENET messages.

ANATOMY OF A PACKET

A packet is a group of files, collected into a compressed archive.  The
standard compression technique defined by this document is ZIP.  Other
techniques such as ARJ, ZOO, ARC, LZH, etc can also be used.  It is also
possible for Unix's tar.Z format to be used to transmit Helldiver packets.
The minimum requirement is a method to collect a group of files into a
single packet, and a method to expand the packet back into the original files.

Each of the filenames in a packet should be stored in upper case on those
systems where case matters.  The following file specifications may appear
in a packet:

	INFO		Optional textual information.
	GROUPS		Index of the newsgroups within this packet.
	*.ART		Text of the articles in a particular newsgroup.
	*.HED		Header information for articles in a newsgroup.

Other filenames may also appear in the packet, but are not defined by this
specification, so they should be avoided by generating software, and ignored
by receiving software.

The INFO file is an optional text file which may contain any kind of
information from the generating system.  Typically this file would only be
present if there is some kind of urgent message that must be sent to the
receiving user.  Use of this file to store the name of the generating BBS
and other such static information is possible, but discouraged to save space
and transmission time.  Lines in this text file are terminated by LF
characters (not by CR-LF pairs).

The GROUPS file contains an index of the newsgroups present within the packet,
specifying the name of the newsgroup and the filename the articles may be
found in.  This is specified further in the next section.

The *.ART files contain the text of the articles from a single newsgroup.  The
actual format of this file depends on the type of newsgroup specified in the
GROUPS file.  See the section "ARTICLE AND HEADER FILES" below for more
information.

The *.HED files provide an index into the *.ART files, specifying where each
article starts and the contents of some of the common USENET header fields.
These files are intended for use by reading software on the recipient's system
to quickly display an overview of the articles present in a newsgroup.  See
the section "ARTICLE AND HEADER FILES" below for more information.

GROUPS FILE

The GROUPS file is a text file containing zero or more lines, each of which
specifies a single newsgroup, its type and the name of the article/header
file pair in which the articles appear in.  In particular, each line has
the following form:

	prefix<TAB>newsgroup name<TAB>type

where "prefix" specifies the name of the article/header file pair, "newsgroup
name" is the name of the newsgroup, and "type" specifies the format of the
articles in the newsgroup.  The article and header files corresponding to the
newsgroup have the names "prefix.ART" and "prefix.HED" respectively.  If
"prefix" contains alphabetic characters, they must be upper case.

The newsgroup name may be any sequence of printable ASCII characters (space
through tilde).  Under USENET, this is typically a dotted name like
"comp.lang.c".  Other networks may include spaces or other unusual characters
in the newsgroup names, so the receiving software must be aware of this fact,
and act accordingly.  Also, receiving software must deal gracefully with
characters that have the high bit set, or names that contain control
characters, since people in other countries that speak a language other than
English may wish to use their country's native encoding for the newsgroup
name.  The only hard rule is that the newsgroup name may not contain TAB,
CR or LF.  Receiving software should treat the newsgroup name as an indivisible
string to be displayed to the user.

The newsgroup type may be one of the following values (case is significant):

	u	USENET news articles
	m	Unix mailbox articles
	M	Mailbox articles in the MMDF format

The individual article encodings are explained further in the next section.
Further types may be defined in future versions of this specification.  If
the receiving software does not recognise a newsgroup type, it should ignore
the corresponding article and header files.

A newsgroup may appear more than once in the GROUPS file, each time with a
different prefix, but this is discouraged.  This could be used to split the
article data across more than one article file, but this is more conveniently
handled by generating a separate packet containing the newsgroup contination.

ARTICLE AND HEADER FILES

The format of the article file depends on the newsgroup type specified in
the GROUPS file.  This version of the specification defines three formats,
which are in common use in the USENET and Unix community.  For each of these
formats, lines are terminated with LF characters.  Any CR characters in
the articles should be considered as data characters, or ignored on receipt.
In particular, MS-DOS systems should strip CR characters from articles before
writing them to a packet.

A 'u' (USENET) article file is a text file consisting of one or more articles
prefixed with an rnews header.  This header has the form "#! rnews n" where
"n" is the number of bytes in the article that follows the header, excluding
the line-feed character which terminates the header.  If the number is
followed by white space and other characters, these other characters should
be ignored, until the next LF character is found.

A 'm' (Unix mailbox) article file is a text file consisting of one or more
articles.  The first line of each article must start with the character
sequence "From ".  Any remaining lines in the article which start with
"From " should have the character '>' prepended.  Thus the "From " lines
delimit the article file into separate articles.

A 'M' (MMDF mailbox) article file is a sequence of one or more articles,
separated by at least 4 Control-A characters.  The article file may optionally
start and end with a sequence of such characters.  If a sequence of 4
Control-A characters occurs in an article, it should be "adjusted" by the
insertion of spaces to split the sequence.  The use of Control-A characters
within an article is discouraged.

The 'm' and 'M' formats were chosen for mail because of their common
occurrence in the Unix community.  The generating system may elect to instead
convert a mailbox into the USENET format if it wishes, and set the newsgroup
name to some descriptive string to inform the reader.

The format of the header files for each of these article formats is the same.
A header file is a text file (LF terminated lines), with one line per article
that occurs in the article file.  The lines in the header file should be in
the same order as the corresponding articles.  Each line has the following
form:

	offset<TAB>subject<TAB>author<TAB>date<TAB>mesgid<TAB>
	refs<TAB>bytes<TAB>lines

[Note: the line-wrapping here is for document-formating purposes only.  No
line-wrapping occurs in the header files].  The fields have the following
semantics:

	offset	Seek position in the article file of where the corresponding
		article starts.  The first seek position is 0.  For the 'u'
		format, this indicates the start of the line following the
		rnews header line.  For the 'm' format, this indicates the
		start of the "From " line and for the 'M' format, this
		indicates the start of the article after the Control-A
		sequence.

	subject	The "Subject:" line from the article.

	author	The "From:" line from the article.

	date	The "Date:" line from the article.

	mesgid	The "Message-Id:" line from the article.

	refs	The "References:" line from the article.

	bytes	The number of bytes in the article.

	lines	The "Lines:" line from the article.  Note that this field
		is pretty useless these days on USENET, but is still popular.
		It is meant to indicate the number of lines in the body of
		the article.  Generating software may elect to re-generate
		this value if it is not present in the original article,
		but this is not required.

If any of these fields contained TAB's or other white space in the original
articles, they should be converted into single spaces.  All fields must be
present, but some may be empty.  The "bytes" field must not be empty, since
it provides necessary information for packet readers.  Each field must conform
to the Internet RFC documents RFC-822 or RFC-1036.

Optionally, a header line may end with one or more extra TAB-separated fields
for other RFC-compliant header fields, together with the header field names.
e.g. "Supersedes: <1234@foovax>".  These fields are not defined by this
version of the specification, and are by arrangement between the generator
and recipient only.

This format is compatible with the newsoverview(5) database format of C-News.
The only difference being the substitution of an offset for the article
number used by C-News.  The C-News format was designed to assist threading
newsreaders, so this packet format should provide similar assistance to
compliant readers.

A note about the rnews header: although a terser separator could be used, the
rnews header has the following advantages: (a) the header files can be
regenerated from the article files if they have become corrupted, and (b) the
article files can be imported into a USENET system as standard rnews batches.
Thus, if the user wishes to set up a real USENET site, or simply use dedicated
USENET software, they can use their existing packet provider as a convenient
read-only newsfeed, with no extra burden placed on the sysadmin of the
generating system.

FUTURE ENHANCEMENTS

The obvious enhancement that can be made is to support other message formats,
especially FidoNet formats.  Currently the newsgroup type code 'q' is reserved
for QWK-format articles.  This will be defined in a future version of this
specification if demand warrants.  Experimentation with other formats is
encouraged, but please contact the author first to prevent double-ups from
occurring.  The author may be contacted via e-mail at rhys@cs.uq.oz.au.

-- 
Rhys Weatherley, Brisbane, Queensland, Australia.   G'day!!
e-mail: rhys@meteor.brisnet.org.au (normal) or rhys@cs.uq.oz.au (long msgs)
Newsgroups: comp.bbs.waffle