From: ERIC@SEARN.BITNET (Eric Thomas)
Newsgroups: bit.listserv.lstsrv-l
Subject: Changes from release 1.7d to 1.7e
Date: 12 Nov 92 02:28:04 GMT
Comments: Gated by NETNEWS@AUVM.AMERICAN.EDU
Comments: To: Revised LISTSERV Maintainers <LSTSRV-M@SEARN.BITNET>


         Description of the changes for release 1.7e of LISTSERV
         -------------------------------------------------------
                           November 11th, 1992

**************
* Highlights *
**************

Version 1.7e introduces the following major enhancements:

- Support for the popular VMSDUMP  format, both for incoming and outgoing
  files. BACKUP savesets  can now be stored as native  LISTSERV files and
  conveniently retrieved in VMSDUMP format.

- Support for MIME  (RFC1341) for outgoing files. Internet  users may now
  order text  files via  e-mail without having  to worry  about truncated
  lines or incorrect ASCII to EBCDIC translation.

- LISTSERV internals,  DISTRIBUTE and inter-server  communication methods
  were changed  in preparation for release  2.09 of the VM  mailer, which
  will have the  ability to handle lines longer than  80 bytes. Note that
  it was  unfortunately not  possible to  test this  code in  a realistic
  setup, and further adjustments might thus be needed before LISTSERV can
  actually use that feature of MAILER V2.09.

- Performance  and robustness  improvements  through the  use of  private
  formatting  routines   rather  than   IBM-supplied  commands   for  the
  generation of files  in Netdata and other NJE  delivery formats (except
  CARD DUMP).  This change makes it  possible to use a  SFS directory for
  LISTSERV's work disk (D-disk).

- Additional improvements  have been  made to  the notifications  sent by
  LISTSERV to list owners such that the proposed commands can be executed
  without  retyping  by  simply   forwarding  the  notification  back  to
  LISTSERV.

****************
* Availability *
****************

Release 1.7 is available free of  charge to sites which qualify under one
(or more) of the categories listed below:

a. Members of CREN or one of the CREN Cooperating Networks, excluding:

   1. Members of the European Academic and Research Network (EARN).

   2. Members whose computing facilities are located in an EEC country.

b. Members  of EARN, CREN or  a CREN Cooperating Network  whose computing
   facilities are located in a nordic country (Denmark, Finland, Iceland,
   Norway, Sweden).

c. Members of  EARN whose computing facilities are not  located in an EEC
   country and  which have formally  applied (in writing) for  a license,
   and  complied  with  the appropriate  EARN  directives.  International
   organizations are not  considered as belonging to  the EEC, regardless
   of where their computing facilities are physically located.

Release 1.6 licenses from EEC countries remain valid for release 1.6, but
will not be carried over to release 1.7.

*******************************
* Supported operating systems *
*******************************

Release 1.7  introduces the  following changes to  the list  of supported
operating systems:

- (1.7a) Withdrawal of support for CMS release 3.

- (1.7a) Support for the CP components of VM/SP release 7 and VM/ESA.

- (1.7b) Support for CMS release 7 in both 370 and XA/ESA/XC modes.

- (1.7b)  Support for  the Shared  File System,  with a  few restrictions
  concerning the  setup of  the A  and D-disks  and the  default filepool
  setting. See  the corresponding section  of the 1.7b release  notes for
  more information.

- (1.7c) Support for CMS release 8 in both 370 and XA/ESA/XC modes.

- (1.7e) Support for the Shared File System for the D-disk.

Support for other versions of CP and CMS is unchanged from 1.6e.

**************************
* External Compatibility *
**************************

Release 1.7e is  generally compatible with 1.7d, from  the perspective of
the  end-user (including  list managers  and maintainers),  and with  the
exception  of the  changes listed  below. Release  1.7e introduces  major
changes  to the  LISTSERV  internals, making  compatibility with  locally
developed extensions  or local  modifications problematic.  These changes
are briefly  described in the next  section, and only affect  sites which
made alterations or additions to  the LISTSERV code. Changes which affect
all  LISTSERV sites  are highlighted  below; note  that more  details are
available, when appropriate, from the longer descriptive section of these
release notes.

 1. The changes to support mail items with lines longer than 80 bytes are
    not transparent for  peered lists if some of the  peers are running a
    version  which does  not support  the  long lines.  More details  are
    provided further down.

 2. The  changes to DISTRIBUTE  to support  mail items with  lines longer
    than 80 bytes can cause problems  with LISTEARN and older versions of
    LISTSERV. See further down for more details.

 3. Release 1.7e  internally uses DISTRIBUTE for all  postings to mailing
    lists, in order to avoid having to  duplicate the code that had to be
    added to support  CRC verification of messages with  long lines. When
    "Mail-via= Direct" is specified, a  new internal DISTRIBUTE option is
    used to  prevent the message  from being forwarded to  other servers,
    but the  message is still  formatted and  delivered to the  mailer by
    DISTRIBUTE code. As a result, the SHOW command no longer displays the
    amount of  postings that were processed  through DISTRIBUTE; instead,
    it shows the total amount of recipients.

 4. LISTSERV now removes control characters from mail messages by default
    and  expands tabs  in the  process. Control  characters usually  have
    undesirable and unexpected results, as they seldom survive the double
    ASCII->EBCDIC->ASCII translation  and cause the  recipient's terminal
    to execute  sequences different  from the ones  meant by  the message
    sender  - not  to mention  the case  of ASCII  control characters  on
    EBCDIC  terminals.  Furthermore,   certain  combinations  of  control
    characters  were found  to create  problems with  IBM's SMTP  (due to
    unexpected CRLF sequences  appearing in the middle of  a record after
    translation).  Lists  which  absolutely  require  control  characters
    must have a "Translate= No" keyword added to their header.

Release 1.7e is  to be installed directly  on top of the  base 1.7d code,
and includes all the fixes available from LISTSERV@SEARN for release 1.7d
(ie  all  the "17Dnnnt  FIX"  files  from  filelist "LFIXES"),  with  the
exception of those containing replacement "recommended user exits", which
always have to be installed manually.

**************************
* Internal Compatibility *
**************************

Major changes  have been made  to LISTSERV internals. The  most important
ones are briefly described below; refer to the LSTSRV-L archives for more
information.

 1. FFORMAT/FCLASS  information is no  longer kept in  GLOBALV variables,
    and must be accessed/changed via calls to LSVCD1.

 2. LSVRECV no longer forces the filemode to D if a value was specified.

 3. The  UUENCODE and  LISTSERV-Punch file  formats are  now provided  by
    built-in LISTSERV code rather than  standalone MODULE's. As a result,
    PDUTIL,  LSVPUNCH  and LSV$UUE  will  be  automatically erased  after
    installation of 1.7e. If you  have local applications which depend on
    them, make a copy before  installing 1.7e. Furthermore, the format of
    SYSFF FILE has changed; if you had installed new file formats on your
    server, you will need to make  a few adjustments. See the comments in
    SYSFF FILE for more information.

 4. Authors of local command packages  should note that LISTSERV does not
    guarantee that the program stack is  in any particular state when the
    command  processor is  invoked. MAKEBUF  and DROPBUF  should be  used
    wherever necessary.

The following REXX files have been converted to PREXX with release 1.7e:

   LSV$UUE   LSVMAIL   LSVSENDF  MIG_*

The following MODULE files have become obsolete with release 1.7e:

   LSVPUNCH  PDUTIL

**************************
* Compatibility Warnings *
**************************

This new section describes planned changes which may affect compatibility
(both external and internal). The release  in which the change is planned
to be  introduced is indicated, with  the letter 'x' denoting  an unknown
release of the specified major  version (not necessarily posterior to the
last release  explicitly mentioned).  Note that  conversion of  more REXX
files to PREXX is assumed to take  place with each new release and is not
indicated here.

- (1.7x) The internal format of  FILELIST/FILEID and attendant files such
  as AFDLIST FILE will change dramatically. Applications which alter them
  directly will no longer work.

- (1.7x) The internal  format of LIST and attendant  files (CACHE, STATS,
  et al) will change dramatically. Applications which alter them directly
  will no longer work.

- (1.7x)  LSVDIAD4,  LSVFRD1, LSVFWR1  and  RXLSVUHF  will be  eventually
  removed. You should avoid using them.

********************************
* Minor fixes and enhancements *
********************************

REMINDER: Manual installation  is not supported for release  1.7, and the
code that  implemented it has  been removed with version  1.7e. Automatic
installation with local  password validation must be  used instead. Sites
which insist on installing manually should note that shipments contain an
installation pre- and post-processing exit. If that exit is not executed,
or if  it is  executed in  a different order  or environment  than during
automatic installation, serious problems may occur.

This  section contains  a brief  description of  all the  "small yet  not
unimportant worries" that have been fixed in release 1.7e, along with the
affected  releases, if  known ("All"  meaning "any  release in  which the
facility  is  available", ie  the  problem  was  present since  the  very
beginning).

Minor enhancements which  do not warrant a detailed  description are also
included; they  are all  located at the  bottom of the  list, and  can be
identified by the word "New" in the "Affected releases" column.

Affected
releases    Description of the bug, problem or enhancement
--------    ----------------------------------------------
  1.7d      Servers running  on VM/SP or VM/HPO  were occasionally unable
            to purge incoming job files  if a severe error occured during
            processing;  the job  would be  reprocessed about  15 minutes
            later and cause multiple error notifications.

  1.7d      LISTSERV will no longer generate addresses of the form:

                  From: "(Joe Smith)" <JSMITH@Somewhere.EDU>

            These  addresses were  not  generated  intentionally and  are
            unnecessarily complex,  but they  are perfectly  valid. There
            exist  popular mail  programs which  cannot handle  them, but
            that only means they are in error.

  1.7a      The recipients counts for network-wide lists produced by LIST
            SUMMARY were  incorrectly calculated. The totals  reported by
            pre-1.7e code happened to be lower than the correct values at
            the time the fix was written.

  All       Some components  of LISTSERV were  unable to append  to files
            with more than  64k records, due to  the unexpected behaviour
            of FSWRITE with RECNO=0 but without FORM=E.

  New       Plural forms  ('Owners' and 'Editors') are  now supported for
            access-level keywords, with the  same meaning as the singular
            forms.

  New       JOIN and LEAVE are now accepted as synonyms for SUBSCRIBE and
            SIGNOFF, respectively, in order to improve compatibility with
            Mailbase.  Mailbase  appears  to   be  the  most  robust  and
            comprehensive  unix-based list  manager currently  available,
            with  the  added advantage  of  not  confusing the  users  by
            claiming to  be a  unix version of  the BITNET  LISTSERV. The
            addition  of  these two  synonyms  will  hopefully remove  an
            obstacle to the use of Mailbase by a larger community.

  New       In order  to make  it easier to  implement "sub-lists"  (a la
            Mailbase) with  LISTSERV, the  syntax of  the "Subscription="
            keyword   has  been   enhanced   to  support   access-control
            specifications,  in  addition  to  the  previously  supported
            keywords ("Open", "Closed" and  "By_Owner"). Sub-lists can be
            convenient when  a rather large  forum is to  discuss several
            sub-topics in  parallel and  participants are  to be  free to
            join and  leave the various  sub-lists as they see  fit. This
            change  makes  it  possible   to  implement  sub-lists  where
            membership is not open.

  New       A side-effect  of the previous change  is that "Subscription=
            Public" is  now a synonym for  "Subscription= Open", removing
            one of the most common mistakes made by new list owners.

  New       LISTSERV no longer needs to swap its work minidisk (192) from
            mode D  to mode Z  and back. This  makes it possible  for the
            work disk to be replaced with a SFS directory, as long as you
            make sure  the directory  is already accessed  R/W as  mode D
            when  LISTSERV  is  started.  Mode  Z  remains  reserved  for
            LISTSERV use (dynamic minidisks, et al).

*******************************************************
* Usability/Performance: Support for new file formats *
*******************************************************

Release 1.7e introduces  support for a number of new  file formats, which
can be  specified on the  'F=' keyword of  commands such as  GET, REVIEW,
etc.

- VMSDUMP, the popular file transfer format for VMS systems running JNET.
  LISTSERV  only  supports  sequential  files. For  incoming  files,  the
  variable portion of  VFC files (usually a time stamp)  is discarded and
  UNDEFINED files are  not supported (but the other 3  stream types are).
  For outgoing  files, LISTSERV only  generates FIXED or  VARIABLE files.
  This  is still  sufficient  to conveniently  store  BACKUP savesets  as
  native LISTSERV files.

- XXENCODE, a version  of UUENCODE modified to use a  more sensible table
  so that the resulting messages survive ASCII to EBCDIC translation.

- Two flavours of MIME base64-encoded messages. Users with a MIME-capable
  user  interface can  use 'F=MIME'  to  order text  files which  contain
  characters that  do not survive  ASCII to EBCDIC translation,  or lines
  longer than the  gateways in the way will  support. LISTSERV translates
  the  text to  ASCII, adds  CRLF sequences,  and sends  the result  as a
  base64-encoded message  of type 'text/appl'. 'F=MIME/APPL'  can be used
  as a  replacement for  UUENCODE to order  binary stream  files (content
  type 'application/octet-stream'). It is important to note that this has
  exactly the  same functionality  as UUENCODE: the  input file  is read,
  ignoring  file attributes  and  record boundaries,  and  turned into  a
  stream of bytes, which are then  encoded and sent. In other words, this
  is not  a suitable method for  transferring text files on  binary VM or
  VMS files.

In addition, release 1.7e no longer uses the IBM-supplied system commands
to generate  files in Netdata and  other formats - with  the exception of
CARD DUMP  format, for which a  modified version of CARD  MODULE is used.
This change improves robustness, performance, and considerably simplifies
functions such as AFD and GET which no longer need to copy files and swap
the work disk around in order to have the right information in the header
or to avoid unwanted side effects of the system commands. The new code is
about 3-4 times faster for both Netdata and DISK DUMP formats.

***************************************************************
* Usability: Support for mail with lines longer than 80 bytes *
***************************************************************

LISTSERV  internals  and  inter-server communication  methods  have  been
changed to support  mail with lines longer than 80  bytes, in preparation
for a new version of the VM mailer with support for Netdata format. There
is no  way to make use  of that support  with the current version  of the
mailer (2.08) and it is perfectly  normal that lines continue to be split
at column 80 after the installation of 1.7e.

These changes introduce a number of problems when communication has to go
through servers not running 1.7e.  Three major components of LISTSERV are
impacted:

1. Inter-server commands: older servers  (including LISTEARN) are able to
   process the new  commands without problem. However,  in certain cases,
   they may  truncate the command if  they, in turn, need  to forward the
   request to other servers. In practice this only affects unlikely cases
   such as e-mail  addresses longer than 80 bytes (the  command itself is
   never  truncated,  but list  of  addresses  passed  in DD's  might  be
   truncated at column 80).

2. Peered lists: a problem will occur if a message with lines longer than
   80 bytes  is posted to  a peered  list and some  of the peers  are not
   running  version 1.7e.  The  message  would be  truncated  as per  the
   following table:

            +---------+-------------------+-----------------+
            | Version | Header truncated  |  Body truncated |
            +---------+-------------------+-----------------+
            |   1.7e  |     Never (*)     |    Never (*)    |
            +---------+-------------------+-----------------+
            |   1.7d  |   Folded at 80    | Truncated at 80 |
            +---------+-------------------+-----------------+
            |  Other  | Truncated at 255, | Truncated at 80 |
            |         | then folded at 80 |                 |
            +---------+-------------------+-----------------+

   (*) If the  local mailer does not support lines  longer than 80 bytes,
       the headers and body will have to be folded at column 80.

   The truncation of the message  text is obviously undesirable, but will
   only happen  if there  were lines  longer than 80  bytes in  the first
   place;  in other  words,  it is  not  an issue  until  mailer 2.09  is
   released. From then on peered lists  will require that all servers run
   release 1.7e.  A temporary keyword,  "Long-lines= No", is  provided to
   assist migration  in cases  where one of  the peers  absolutely cannot
   install 1.7e in time (note that peered LISTSERV-LISTEARN lists are not
   formally supported, but  can benefit from that migration  aid while it
   is being provided). When "Long-lines= No" is specified, LISTSERV folds
   text lines longer than 80 bytes  to bypass this problem at the expense
   of usability.

   The truncation of header lines at  column 255 is potententially a very
   serious problems. Long  header lines may be present  regardless of the
   version of the  mailer being used, as LISTSERV  must internally unfold
   RFC822 fields in order to process them and does not re-fold them until
   necessary  (and since  inter-server jobs  support lines  of up  to 64k
   bytes,  it never  turns out  to be  necessary). Although  header lines
   longer than 255 are rare, this  new mechanism can create problems with
   peered lists unless  all the peers run 1.7d or  1.7e. Because there is
   no problem with the previous version  and older versions are no longer
   supported, no migration aid was provided.

3. DISTRIBUTE: LISTSERV  now uses Netdata format for  DISTRIBUTE jobs, so
   that long mail  lines (up to 64k  and thus well beyond  the 1000 bytes
   SMTP standard)  can be  supported. While  older servers  are perfectly
   capable of  processing such jobs,  the same truncation problems  as in
   point 2 exist. Fortunately, version  1.7e is being released before the
   new VM mailer and it should be possible to remove the few non-LISTEARN
   sites which are  still running 1.7d from the backbone  when the mailer
   is  released, thus  solving the  problem for  non-LISTEARN sites.  The
   structure of  the LISTSERV backbone  in Europe is such  that countries
   running LISTSERV will seldom be affected by truncation due to LISTEARN
   sites, so  this is fortunately  a private LISTEARN problem  which will
   have to be solved separately.

IMPORTANT:  The use  of  Netdata for  DISTRIBUTE  will cause  significant
performance problems  to sites running  a version older than  1.7d. There
will be  a noticeable impact for  1.7c sites and  a factor of 2  or worse
degradation for  1.7b and older.  The impact of  this change on  1.7d and
1.7e sites is negligible, and, for  1.7e, it is more than compensated for
by other performance changes to DISTRIBUTE.

***************************************************************
* Usability: Simpler administative procedures for list owners *
***************************************************************

The 'Subject:'  field conventions for administrative  messages introduced
with version  1.7d have proved to  be very popular with  list owners, and
are being generalized with 1.7e.  Furthermore, messages which request the
list owner to confirm a certain action  (such as adding a user to a list,
or changing  his distribution options)  are now specially  formatted such
that  simply forwarding  the  message  back to  LISTSERV  will cause  the
command to be  automatically executed without having to  be retyped. Note
that the message  must be forwarded, and not replied  to. While there are
user  interfaces  whose reply  function  will  cause  the command  to  be
properly executed, in the majority of cases this will not work (even with
REPLY TEXT). Please be careful not  to suggest that users use REPLY, even
if it happens to work for you.

**************************************************
* Usability: Welcome message for new subscribers *
**************************************************

Release 1.7e no longer requires a change to the list MAILFORM in order to
provide a "welcome  message" for new subscribers. The  main problems with
the MAILFORM solution  is that postmaster intervention  is required every
time the message has to be updated,  for security reasons, and that it is
very easy to  make mistakes as MAILFORM files are  actually programs with
(a majority of) imbedded text. In  particular, it is very easy for novice
list owners to forget to double the quotes.

Once a subscription has been accepted,  LISTSERV will look for a LISTSERV
(not CMS) file called 'listname WELCOME'. This is a plain text file which
can be easily created and maintained  by the list owner if the postmaster
has created a  filelist for the list.  The first line, if  it starts with
'Subject:', defines the  subject field of the  welcome message; otherwise
LISTSERV uses  a pre-defined subject and  uses the first line  as part of
the message text. Again, contrary to the MAILFORM's, the welcome messages
are  LISTSERV  files;  simply  placing   a  'listname  WELCOME'  file  on
LISTSERV's A-disk will not accomplish anything.

Since this  is a plain text  file, there is  no need to double  quotes or
include formatting commands for text which must not be reformatted. This,
of course,  means that MAILFORM  variable substitution sequences  are not
available.

*********************************************************
* Usability: LB64.C - a tool for non-BITNET list owners *
*********************************************************

LB64 is a  new tool which can  be used by non-BITNET list  owners to send
files or  commands (and in  particular PUT requests) to  LISTSERV without
having to worry  about lines longer than 80 bytes  or characters which do
not get  translated properly  by the gateways  the message  goes through.
LB64 takes advantage of a  server-to-server protocol which was added with
release 1.7e  but was not designed  to solve the problem  of PUT requests
from the  Internet with  lines longer  than 80  bytes (this  problem will
disappear when  the VM mailer  supports Netdata format). In  other words,
LB64  does not  claim  to be  anything  but a  useful  side-effect of  an
unrelated LISTSERV  development; if you dislike  it, you are free  not to
use it.

The program is written  in C and is designed for  unix systems, where its
output can be  conveniently piped to a  'mail listserv@hostname' command.
It can however be used on any system that can emulate unix text files, be
it ASCII or EBCDIC. For more information, issue a 'GET LB64.C' command to
LISTSERV and read the comments in the program.
Comments: Gated by NETNEWS@AUVM.AMERICAN.EDU