From: michael@gandalf.moria (Michael Haardt)
Subject: [documentation] wnews.1 -- wnews overwiew manual
Keywords: wnews
Reply-To: u31b3hs@pool.informatik.rwth-aachen.de (Michael Haardt)
Organization: An old and gray machine, somewhere in Moria.
Date: Mon, 14 Sep 92 23:07:18 +0100

Ok ok ok before I get flooded with requests and questions, I will post 
the general WNEWS 3.00 manual.  I really wonder why you don't have it
already, but the questions I got seem to indicate that it is indeed
missing, and the number of people who asked questions about wnews
seems to justify the posting.  Note that the manual is not formattable
with real nroff, only with that weird MINIX nroff thing, so better
having the sources than a screwed up formatted copy.  A try can't hurt,
because I don't know Coherent's nroff.

If you want to write an obsolete flame about wasting bandwidth ... don't 
post.  It would be a waste of bandwidth too, and people will not even 
archive your flame!

Michael

PS: Do I need to say that you can get the complete MINIX wnews version
    from the well known MINIX archives?  I think that is obvious.
-------------------------snip,snap -- FvKs good stuff-------------------------
.\"
.\"      MicroWalt Corporation/MINIX User Group Holland (NLMUG)
.\"
.\"                 W-NEWS NEWS Processing Package
.\"
.\"                      Fred N. van Kempen
.\"
.\"                  Version 3.00 - 03/30/91
.\"
.\"
.\"     Format this document with the command "nroff -man name"
.\"
.\" Document-specific macros

.\" Indent a line.
.de IN
.ti +5
.nf
\fB\$1\fP
.fi
.br
..

.\" Define a chapter.
.de CD
.bp
.sp 7
.nr A +1
.nr B 0 0
.ce
.nf
\fB\nA.   $1\fP
.fi
.sp 3
..

.\" Define a section.
.de HD
.ne 5
.sp 2
.nr B +1
.nf
\fB\nA.\nB  $1\fP
.fi
.sp 1
..

.\" Document global settings
.ll 72
.in 8
.nr A 0 0
.nr B 0 0

.\" Title page
.bp
.sp 15
.ce 5
.B "MicroWalt Corporation/MINIX User Group Holland (NLMUG)"
.sp 2
.B "W-NEWS NEWS Processing Package"
.sp 2
.B "Fred N. van Kempen"
.sp 2
.B "Version 3.00 - 03/30/91"
.sp 2
.B "R E F E R E N C E    M A N U A L"
.sp 2
.sp 10
.B ABSTRACT
.br
W-NEWS is a NEWS processing package which resembles the
.B B-News
package found on many UNIX systems. It was written especially for
use with the MINIX operating system, but it will work on most other
UNIX-like systems as well, since virtually no system-dependent code
is used by the programs.
.bp 1
.\" Header and footer lines.
.he "MicroWalt W-NEWS""Reference Manual
.fo "Version 3.00(Beta)""Page %

.CD "I N T R O D U C T I O N"
.B W-NEWS
is a set of programs that can be used to process news articles,
both incoming and outgoing.  This Reference Manual describes its
bugs, shortcomings, workings and history.

.HD "THE OLD DAYS"
I have been running
.B UUCP
on MINIX for some years now, and with success, as is known by
many people. I wrote some application programs such as
.B W-MAIL
and
.B U-MAIL
that let me communicate with the rest of the world in a fairly
convienient manner, and I am very pleased with it all.
.sp 1
However,
.B NEWS
access has always been a bother. My UUCP host did not have a news
feed itself, and I got somewhat tired of dialing in on my former
school to read news there. Finally, my account expired, and I was
left with no news access at all...
.sp 1
Luckily, about half a year ago someone offered to send me articles
from
.B comp.os.minix
by e-mail (note, that I did have a more-or-less reliable mail link),
which I gratefully accepted. It worked fine, but it took me lots of
time to process all incoming articles. Also, a newsreader was not
present...
.sp 1
So, to ease the reading of those articles, I ported the
.B vn
newsreader (posted by Tony Andrews) to MINIX, with only little
effort. This cured only part of the problem, since I still had to
insert the incoming articles into the NEWS spool directory...
.sp 1
About three months later, I got a "normal" UUCP link, and the system
manager of that host offered to send me comp.os.minix as
.B "batches"
of articles. I didn't have a clue as to what those things were, but
I told the guy to send 'em anyway...
.sp 1
While experimenting with the incoming data, I gradually found out
what a news batch was, and how it was constructed. I wrote two
little programs (
.B rnews
and
.B newsrun
) that allowed me to automatically process incoming articles.
.sp 1
The first stripped off this funny-looking
.sp 1
.IN "#! cunbatch"
.sp 1
header, fed the remainder through compress(1) and stored the result
in a spool directory for later processing by
.B newsrun
, which was started daily by cron(1). Newsrun unpacked the decompressed
batches, and stuffed all articles into the
.B /usr/spool/news/comp/os/minix
directory, just like vn wanted it. It also maintained an
.B active
file, because vn wanted it to exist. This worked well enough to keep
me satisfied until I set up the
.B MUGNET
network.

.HD "MUGNET and B NEWS"
A network backbone needs much more robust software than the little
package I wrote, and this I realized very quickly after starting up.
My package could only handle one group (it stored all incoming articles
in one directory!), and posting news was impossible (besides mailing it
to some relay system).
.sp 1
So, since we now had to handle news groups from lots of hosts, plus a
number of MUGNET groups, I had to use a package smarter than my own.
I dislike
.B "C News"
because it depends on shell scripts too much (which is
.B "very slow"
on MINIX !!!). Therefore, I finished the 'cleanup' of the MINIX-port
of the
.B "B News 2.11"
software that Peter Housel sent me, and I started using that on the
MUGNET-Europe backbone.
.sp 1
However, although my machine is large enough to run 'B', many small
MINIX systems around me (in MUGNET) cannot handle such a beast. We
just needed a simple version of the 'B' software which does not try
to do it all, since many of the special features offered by large
packages like 'B' use either special UNIX-ish tricks to get things
working, or they can be lived without (for example, the support for
old-style news articles).

.HD W-NEWS
All of this lead to me starting a major rewrite of my old NEWS package,
under the name of
.B W-NEWS
(what else? :-). Since I like the way 'B' handles things, I decided to
try and give W-NEWS as much of the 'B' look as possible. It now consists
of the following modules:
.sp 1
.IN "rnews     - handle incoming articles and batches"
.IN "unbatch   - process spooled news"
.IN "inews     - the main article relayer"
.IN "control   - implementation of control messages"
.IN "sendbatch - create batches of articles for transport"
.IN "expire    - remote old articles from the system"
.IN "pnews     - post a news article"
.IN "checknews - check if any new articles have arrived"
.sp 1
As can be seen, it is a fairly complete system. Incoming articles or
batches of articles are moved to a NEWS spool directory by the
.B rnews
program for later processing (cron!) by
.B unbatch
(also callable as
.B "rnews -U"
). Unbatch calls
.B inews
for each article to be posted locally. The latter also takes care
of the forwarding (relaying) to other systems connected to the host,
which is required for backbone operation. Inews is also called by the
.B pnews
news posting program (a derivative of the
.B W-MAIL
Message Editor) for the actual posting (both local and remote) of the
article. Posting is achieved by putting a copy (well, actually a link,
since each article is put on disk only once; all other copies in other
news groups are links) in the desired subdirectory of the NEWS spool
space. For any article forwarded to a certain remote host, a line
containing the full pathname of the article is added to a text file in
the
.B /usr/spool/batch
directory. The
.B sendbatch
program (which should be executed at least daily by cron) creates
batches (optionally compressed) from these outgoing articles, ready
for transmission with a network protocol, usually UUCP.
So, the batches end up in the UUCP spool directory for transmission
to the remote system.
.sp 1
This setup allows you to both receive (with
.B vn
) and post (with
.B pnews
) news articles, for multiple news groups the same way 'B' does it.
.sp 1
For a newsreader, I recommend the
.B vn
program, or, if you don't like using full-screen software, the
.B rna
(also called
.B readnews
)
program. These are both available from the NLMUG Archive Service. There
will be a simplified version of the vn program in the W-NEWS package in
the future.
.sp 1

.HD "OUTLINE OF THE REST"
The rest of this document will describe all the modules of the W-NEWS
system, one per section.  Following that, a description of all files
used is presented.

.CD "M O D U L E S"
This chapter describes the inner workings of the various programs
(modules) of the W-NEWS system.  It does not, however, tell you how
to invoke the programs.  See the respective manual pages for that
information.
.HD rnews
Usenet (UUCP protocol) or Internet (TCP/IP protocol).  They come
in as simple datafiles.  When these datafile are ready to be
processed by the NEWS system, the networking software calls the
.B rnews
program to take care of processing the newly received data. In
many case, however, it is at least undesirable to have the system
process the data right away, since this processing may cause the
system to be slowed down quit a bit.  Therefore, this program only
moves the datafiles to a spool directory of the NEWS system.  By
default, this is the
.B /usr/spool/news/.rnews
directory, and the filename of the datafile will be something like
.sp 1
.IN "yymmddhhmmXXXX"
.sp 1
with
.B yy
being the year (minus 1900),
.B mm
the month (01 is January),
.B dd
being the day of the month (1-based), and
.B hh
and
.B mm
being the time in hours and seconds respecively.  The last four
characters of the name are the process ID of the rnews program
in hexadecimal (to make them fit in 14 characters total).
.sp 1
This file will later be processed by the
.B unbatch
program.
.sp 1
After the file has been queued, rnews exits.  See the manual page
for
.B rnews(8)
for more details on how to invoke this program.
.HD unbatch
At regular time intervals, the system should start the
.B unbatch
program, to process the news that came in that day.
.sp 1
The first thing unbatch does is check if there is already a copy of
it running on the system.  This is indicated by the presence of the
lock file called
.B .unblock
in the news LIBDIR (usually
.B /usr/lib/news
).  If it exists, unbatch simply exits.  If the lock file does not
yet exist, unbatch creates it, to indicate that there is now a copy
of unbatch running.
.sp 1
Incoming news can be in either one of three forms:
.sp 1
.IN "1. Regular articles."
.IN "2. Batches of articles."
.IN "3. Compressed batches of articles."
.sp 1
All of these are expected to be in the "incoming" spool directory
of the NEWS system, usually this is the
.B /usr/spool/news/.rnews
directory.  When this program is invoked, it will search the directory
named above for files, which it then processes one at a time.
.sp 1
Each file is first opened.  Then, unbatch will read the first line of
text from the file (up to 128 characters in length) to determine the
type of this file.  The folllowing rules apply:
.sp 1
.B "COMPRESSED BATCHES"
are files with a line of
.sp 1
.IN "#! cunbatch"
.sp 1
as their header.
.sp 1
.B "BATCHES"
are files with the text
.sp 1
.IN "#! rnews NNNNN"
.sp 1
as their header.
.sp 1
.B "ARTICLES"
are all other files.
.sp 1
First of all, unbatch checks if the file is compressed.  If so, it
executes a command line of the form:
.sp 1
.IN "/usr/bin/compress -d"
.sp 1
with the standard input being the current input file, and with the
output being a temporary file in the
.B /tmp
directory.  After successful decompression (indicated by an exit
status of 0), unbatch closes the current file, and opens the newly
created temporary file as the current input file.
.sp 1
At this point, it can either expect to get a batch of concatenated
articles, or a plain, single article.  To see what type of file it
has currently open, it continues the check with comparing the first
9 characters of the line with the text:
.sp 1
.IN "#! rnews "
.sp 1
since this indicates the start of an article in a batch file.
.sp 1
Unbatch now goes into a loop, extracting all articles in the current
open file and posting them by calling the
.B inews
program repeatedly.  It does this by copying each article into a
temporary file, and then issue a command like:
.sp 1
.IN "/usr/lib/news/inews -h"
.sp 1
with standard input being the temporary file containing the article.
The inews program then takes care of the actual posting of the
article.
.sp 1
When a file has been completely processed, it will be removed if no
errors occurred during the unpacking.  Otherwise, it will be moved
to the parent directory, to allow the system administrator to unpack
it manually.
.sp 1
When all files in the directory have been processed in this manner
(relax; this may take some time... :-) the lock file is removed and
unbatch exits.  See the manual pages of
.B unbatch(8)
and
.B rnews(8)
for more details on how to invoke unbatch.
.HD inews
.B Inews
is the heart of the NEWS system.  It receives articles from the
unbatching program for local posting (and possibly forwarding to
other systems), and it receives articles that are generated by
news composing programs like
.B pnews
or news readers like
.B vn
when a followup is created.
.sp 1
When inews is started, it reads the article's header (as far as one
exists) into memory.  The rest of the article is copied into a
temporary file in the
.B /tmp
directory.  Now, inews checks if it has to generate a message header
(which is the case if the article was generated locally), or if it
only has to update the information in the header it just read into
memory (which is the case when unbatch calls inews with an article).
A message header usually has the following fields in it:
.sp 1
.nf
  * Path:
  * From:
  * Sender:
    Reply-To:
    Followup-To:
  * Newsgroups:
    Distribution:
    Control:
  * Subject:
    Keywords:
    Summary:
    References:
  * Message-ID:
    Organization:
    Department:
  * Date:
    Expires:
    Approved:
    Lines:
.fi
.sp 1
All fields marked with an asterisk (*) are mandatory.  The other
fields need not be present, although they frequently are.  Some
fields (From, Sender, Reply-To, Subject, Message-ID, Organization,
Date) have the same meaning as in e-mail message headers.  The rest
are special to the NEWS system.
.sp 1
As soon as inews finishes updating the article header, it creates
a complete article (comprised of the new article header plus the
article body) in a file in the NEWS main spool directory (usually
this is
.B /usr/spool/news
).  Now, a note is written to the NEWS system log file about the
arrival of this article.  If resource accounting has been enabled,
inews will also write a record about the arrival to the NEWS
accounting file, after which the temporary file in /tmp is deleted.
.sp 1
Inews will now check the system's
.B sys
file to see if the local system is interested in this article.  If
it is not, the system will try to put the article in a news group called
.B junk
, which is used to dump all unwanted articles in.  Then, it checks
to see if the directory for this news group exists, since it has to
be put somewhere.  If no directory exists, inews will change the
news group name to
.B junk
again.
.sp 1
If the
.B HISTORY
mechanism was enabled at compile time, inews will now check its
history database to see of this article has already been received
before (which is, in theory, possible if a system has more than one
news feed).  If it already got it before, it will write a message to
this effect in the log file, and stop the delivery. Otherwise, a
record for this article will be added to the history database.
.sp 1
Inews then scans through the system's
.B active
file, to see which sequence number the article will get.  If the
news group cannot be found in this file, inews will try to use
the junk group again, to prevent the article from getting lost.
.sp 1
Inews now locks the active file, and increases the number of the
next available article.  Finally, the article's temporary copy
is linked to a file with a name identical to the ASCII representation
of the sequence number for this article.  So, when using a news
group of
.B comp.os.minix
, we get a total path name of:
.sp 1
.IN "/usr/spool/news/comp/os/minix/NNNNN"
.sp 1
with
.B NNNNN
being the number of the article in that news group.  This naming
scheme allows for up to 65535 articles per news group, without the
physical need for expiration of articles.  Larger NEWS installations
use a 10-digit number instead of the usual 5-digit number.
.sp
This process of linking the temporary copy of the article to files
in the spool directories of the various news groups is repeated for
all news groups in the
.B Newsgroups:
field of the article.  Thus, only one copy of the article is on the
disks; the rest of all occurrences are links to the same article
to save disk space.
.sp 1
.B NOTE:
this requires all directories to be on the same disk!
.sp 1
Finally, inews will check the
.B sys
file to see if it has to send this article to other systems as well
(this is called
.B relaying
of news articles).  If so, it will send the article to the remote system
as described in the
.B FILES
chapter under section
.B sys
, since this can be set on a per-system basis.
.sp 1
Again, a note is written to the system log file telling to which systems
the article has been relayed.  Also, if resource accounting is enabled,
a record is written about the relaying of this article.
.sp 1
Inews now has successfully posted the incoming article to the local
system, and to any systems that were interested in the article.  It
will remove all temporary files, and exit.
.sp 1
For more information on how to invoke the inews program, see the
.B inews(8)
manual page.
.HD control
This program implements the various types of
.B "control messages"
that can be sent out using the NEWS system.  Typically, these
messages are used to add and/or remove news groups, and to
cancel previously sent articles.  Since most of these messages
share a fair amount of code, it was implemented as a single
program with a link for each of the available functions.
.sp 1
A control message is a news article with a special field in the
message header (named
.B Control:
), which can be used to tell other NEWS-running systems to perform
certain management-operations like creating or deleting a newsgroup,
or cancelling a message that was previously sent.
.sp 1
For example, the system administrator of a network may decide to
create a new newsgroup on the network.  He (or she) then posts a
control message, which looks something like:
.sp 1
.nf