Newsgroups: bit.listserv.xmailer
Date:         Tue, 29 Sep 1992 01:46:12 +0100
From:         Eric Thomas <ERIC@SEARN.BITNET>
Subject:      Re: File Forwarding to Internet Nodes
              28 Sep 1992 16:10:41 CDT from The Crosswell Mailer List
              <XMAILER@FINHUTC>

On Mon, 28  Sep 1992 16:10:41 CDT "(David  E Boyes)" <dboyes@IS.RICE.EDU>
said:

>base64  is completely  printable stuff,  and carries  the character  set
>along with it.

This is correct: as  long as the character set is ASCII or  one of its US
variants,  it can  be specified  in  the header.  EBCDIC is  not a  valid
character set name, and the definition  of new character set names (which
must  be   registered  through  the  IANA)   is  explicitly  discouraged.
Furthermore, the RFC  mentions EBCDIC many times, so this  does not sound
like an oversight. Nor am I surprised.

>You could probably use Eric's LB64.C code as a start, although something
>like this should probably be in assembler for efficiency.

LB64.C  is  designed to  transfer  job  files  which may  contain  binary
characters and may originate on ASCII  or EBCDIC systems. However it does
not carry such  information as date, filemode number,  record format, and
so  on. In  other words,  it will  work for  RECFM V  files and  will not
preserve the date; something more  sophisticated would be needed for full
support of CMS files.

The LISTSERV file  formatting routines are written in PASCAL,  and if you
have ever  examined the code VS  PASCAL generates you will  know how much
slower it  is than assembler  (not to mention the  fact that you  have to
make  a function  call  to  do pointer  arithmetics).  The compiler  will
happily store a register  to a work area, and then  reload the value from
the work area into another register  and use that second register just to
compare it  to a  constant. Accessing  the first element  of an  array is
generally  performed  by getting  the  address  of  the array  plus  one,
subtracting 4096 as a matter of routine, and then adding 4095 to get back
to  square one.  That is  still  some 2-5  times faster  than the  native
assembler commands.

>I'd probably oppose using an application/netdata type, because it's just
>too hard  to handle  on non-IBM  hosts. The CARD  format is  much better
>documented,  and commonly  available, at  least on  VM systems.  It'd be
>relatively easy  to write CARD  client programs for other  OS'es, rather
>than have to get IBM to release  the NJE spec to the world w/o licensing
>fees.

I have to disagree. First, the CARD format is NOT documented, you have to
read the source if you want to  know. NETDATA is documented in one of the
ESA 1.1  manuals. But that is  not the most serious  issue. While writing
code that  READS a CARD deck  on a non-VM  system is fairly easy,  I have
decided, after  writing half of  the code  for nothing, not  to implement
CARD support in the LISTSERV file  formatter. CARD LOAD will crash if you
don't feed  it the blocksize and  number of blocks of  the original file;
furthermore this information  is used in an  obscure internal calculation
to determine if  there is trailing data  to trash. It would  have been no
problem to finish the CARD code in  LISTSERV, but then it would only work
when the input  is an actual disk file,  or a RECFM F file  for which the
values can be calculated. With data  from a DISTRIBUTE job it would never
work without staging  it to a disk  file, and you would  need to simulate
the VM file system structure on non  VM systems. It is not very difficult
but, added to  the fact that you  have to know the number  of records and
LRECL *in advance*,  even for RECFM V  files, it was enough  to make CARD
the  one format  that  LISTSERV  will only  support  through an  external
interface (CARD MODULE).

Netdata  is  a  royal pain,  it  required  a  lot  more code  than  CARD,
especially if you  want to do things decently. Above  all it is extremely
frustrating to have to spend  lots of programming time encoding mandatory
fields which contain garbage, and in very unpleasant formats to boot. But
you don't need  to know the number  of records in advance,  nor the exact
LRECL  (except  for fixed  format  files):  the  LRECL  is only  used  to
dimension a buffer. When decoding, it is very easy to ignore the crap you
don't want  to hear  about, but  spanned records  are not  convenient and
writing code that won't go into a loop  if the input file is invalid is a
pain.

DISK is easiest, but it should not  even be considered. It was designed a
VERY long  time ago  for ease of  sorting should your  cards fall  to the
floor and  get mixed  up. 30  columns out  of 80  are rubbish  - sequence
number, "DISK  DUMP" identifier, a copy  of the fileid on  each and every
card, and so on.  When dumping, you only need the  file statistics at the
end, so  you can conveniently determine  them yourself in one  pass. When
loading, you have to do two passes for RECFM F files to get the LRECL.

Believe it or not, VMSDUMP would be my choice if I had to pick one. But I
think it is easier to just design a new one.

  Eric
Comments: Gated by NETNEWS@AUVM.AMERICAN.EDU