Newsgroups: bit.listserv.lstsrv-l
Date:         Tue, 6 Oct 1992 21:13:58 +0100
From:         Eric Thomas <ERIC@SEARN.BITNET>
Subject:      New LISTSERV release planned - preliminary announcement

I am planning to release a new version of LISTSERV soon. One reason is to
take care of the whining from PROFS users and those unix users whose mail
program cannot parse fields of the type

                     From: "(Joe Doe)" <JOE@DOE.EDU>

(admittedly  ugly and  definitely not  intentional, but  certainly not  a
violation of any Internet mail header standard I know of). A more serious
reason is  to package all the  minor 1.7d fixes together  for sites which
don't know  or want  to use  LFIX. But  the most  important reason  is to
introduce significant changes to LISTSERV  internals and to DISTRIBUTE in
order to support the coming Netdata-capable VM mailer. This may seem like
a trivial issue, but  it is more work than the  new BITEARN NODES format.
You have to do something about each  and every place where a line of text
is punched to the  mailer. You have to do something  about each and every
place where  the CMS stack is  used to transfer data  between EXEC's when
that data may end up being sent to the mailer. The "something" is usually
not obvious,  especially given the  performance issues. That is  the easy
part. Then you  have to modify DISTRIBUTE et al  to generate inter-server
jobs with  longer lines, which  would be trivial  if you would  accept to
sacrifice RSCS  list processor support  and to dump all  pre-1.7e servers
from the backbone (after some  "transition" version which would introduce
support on the  receiving end while still generating on  the old format).
Then you have to think about what happens to older servers when you start
sending them  lines longer than  they expect,  and that is  the difficult
part.  For instance,  a new  CRC mechanism  had to  be designed,  since a
server that is programmed to fold lines at 80 in order to give the mailer
a file  it can handle  will not  get the same  results as a  server which
doesn't fold.

In spite  of all these  changes, 1.7e still  won't have full  support for
long mail lines. One  reason is that there are areas where  a lot of work
remains to be  done, and where one  has to worry about the  impact on end
users  of suddenly  turning  on  the generation  of  long  lines while  a
significant  amount of  servers/mailers  still only  support 80  columns.
Another reason  is that zilch information  about the new mailer  has been
released, so for instance I have no idea how to ask the mailer whether it
can handle NETDATA files and automatically enable this, and of course the
code has never been tested.

For  now,  1.7e  still  folds  messages sent  to  mailing  lists  at  80.
DISTRIBUTE and most internals can handle  lines of up to 16M (in practice
limited to 64k by the network  transfer formats and/or file system limits
for  RECFM V  files). Most  other mail-based  functions have  a limit  of
either 16M/64k or 255, in some cases  I decided to still fold at 80 until
there  is a  majority of  systems supporting  long lines,  even though  a
higher limit is technically possible. That is, the only function which is
really limited to 80 bytes is the  main mailing list code, because I have
to  study the  impact  on older  servers before  I  change anything,  and
because the stack is used all over the place.

The remainder of  this note is mostly for beta-test  sites, who can start
applying now,  even though I  haven't yet  decided whether to  start beta
testing  tomorrow  or next  week  (it  will  depend  on how  much  damage
LISTSERV@SEARN turns out to do to older servers during alpha testing). To
make life  simpler, please sign  off the beta-test  list if you  had beta
tested 1.7d and cannot do it with 1.7e.

The most  severely impacted function  is DISTRIBUTE. Netdata is  now used
for inter-server jobs,  so that older servers can  process jobs generated
by  the  new  servers  without  change. This  will  introduce  a  serious
performance degradation  for 1.7c  and earlier  versions, I  apologize to
LISTEARN sites but there was no other workable solution. The new jobs may
contain a comment card of random length  in the middle, this is not a bug
but a  "filler" line to  align the Netdata stream  on a card  boundary so
that the  RSCS list processor can  be used. The Netdata  headers on these
jobs are a minimal set - only  the fields required by the standard. This,
and the fact  that they are generated  on the fly, makes  them look weird
when PEEKed. With CMS8 (and maybe CMS7), you will be told you haven't got
enough storage to view  it, and then a bunch of  stacked commands will be
executed; there is an APAR for that, and a circumvention is to use a very
large virtual machine  (12M, or then 30M). Assuming PEEK  doesn't trip on
the bug, the reported filename will be "*............................" or
something  equally  unexpected. Again,  this  is  none  of my  doing  and
definitely not a bug in LISTSERV,  DMSDDL just has a stupid default value
for an  optional header field,  and no I  won't bother fighting  this one
with IBM, especially as I don't use PEEK myself.

1.7e will mark DISTRIBUTE jobs  with another CRC signature, '2-xxxxxxxx',
which older servers should ignore but forward on. 1.7e can check both old
('1-xxxxxxxx') and new CRC's, and unfortunately  it will have to let jobs
with an  error on the new  CRC through if  the old CRC matches,  as older
servers are  likely to fold/truncate  jobs with  long lines and  you then
have a choice  between accepting them or sending them  to the postmaster,
who will either resend them (same  results) or throw them away. When this
happens,  a message  will be  printed on  the console  but NOT  mailed to
anyone. This is  not an error, it  means two 1.7e servers  talked to each
other through an old server which folded the lines.

BSMTP  is now  used  for delivery  of all  DISTRIBUTE  messages, both  to
simplify the new code and for fear that the BITNET 2 exit might not fully
support Netdata and/or long address  lines. Unfortunately, this will slow
down the mailer since BSMTP header processing is much slower than the old
BITNET 2.  Delivery to local recipients  will be faster with  1.7e due to
other changes, and the combined effect  is that the mailer might build up
a queue much faster than before, so you will want to watch out for that.

  Eric
Comments: Gated by NETNEWS@AUVM.AMERICAN.EDU