X-Mailer: ELM [version 2.3 PL11]
Newsgroups: bit.listserv.xmailer
Date:         Tue, 29 Sep 1992 09:57:24 CDT
From:         "(David E Boyes)" <dboyes@IS.RICE.EDU>
Subject:      Re: File Forwarding to Internet Nodes
Comments: To: XMAILER%BITNIC.BITNET@ricevm1.rice.edu
              92 1:46 am

I said:
> >base64  is completely  printable stuff,  and carries  the character  set
> >along with it.

Eric Thomas said:
> This is correct: as  long as the character set is ASCII or  one of its US
> variants,  it can  be specified  in  the header.  EBCDIC is  not a  valid
> character set name, and the definition  of new character set names (which
> must  be   registered  through  the  IANA)   is  explicitly  discouraged.
> Furthermore, the RFC  mentions EBCDIC many times, so this  does not sound
> like an oversight. Nor am I surprised.

I understand that a ISO 10646 character set option is under
discussion. Since ISO 10646/Unicode II covers just about every
character encoding in the industrialized world, and most of the
others too, this might be an acceptable way to look at it. That
way everyone will at least be able to point to one encoding
standard instead of figuring out which EBCDIC variant we're
using. (I can hardly wait to send a file to someone who uses
Modified Urdu!)

As a side not, isn't Internet mail defined to be streams of ASCII
characters delimited by CR/LFs? If so, then I can see the basis
of the MIME restriction.

I said:
> >You could probably use Eric's LB64.C code as a start, although something
> >like this should probably be in assembler for efficiency.

Eric said:
> LB64.C  is  designed to  transfer  job  files  which may  contain  binary
> characters and may originate on ASCII  or EBCDIC systems. However it does
> not carry such  information as date, filemode number,  record format, and
> so  on. In  other words,  it will  work for  RECFM V  files and  will not
> preserve the date; something more  sophisticated would be needed for full
> support of CMS files.

I realize that LB64.C wouldn't handle the general case -- that's
why I said it would be a *start*. You would have to combine the
MIME encoder with some form of packaging tool like CARD or
something supporting Netdata to handle the operating
system-dependent stuff. What I was thinking of would be an inline
encoder that packaged the file to be forwarded line by line into
CARD or Netdata format and then formatted it into MIME-compatible
octet streams as the output stage. Once in that format, it could
be safely mailed to another system and decoded there.

Eric said:
> Accessing  the first element  of an  array is
> generally  performed  by getting  the  address  of  the array  plus  one,
> subtracting 4096 as a matter of routine, and then adding 4095 to get back
> to  square one.  That is  still  some 2-5  times faster  than the  native
> assembler commands.

I'm a bit confused by what you mean here. Faster in terms of
development time? If so, then yes, I agree. If you mean faster in
the general case, I think that something like this in a heavily
used production service will be called enough times to warrant
the extra time spent to hand-tweak the assembler into submission,
especially if your users are anything like mine and tend to get
complete models of the entire Earth's magnetic field strengths from
various national laboratories on a regular basis via SENDFILE and
want them forwarded to their Vaxen for more groveling...8-).

Eric said:
> I have to disagree. First, the CARD format is NOT documented, you have to
> read the source if you want to  know. NETDATA is documented in one of the
> ESA 1.1  manuals.

My point was less technical than logistical. IBM is unlikely to
release the specs for Netdata into the public domain without
charging some form of licensing fee. I'm not married to the idea of
using CARD, but it has the advantage of being already widely
available and free of annoying legal hassles. In these days of
software "look-and-feel" lawsuits, it's a very real bogeyman. I
know *I* can't afford legal help that could seriously offer any
challenge to IBM Legal on *any* point, and I doubt Princeton or Rice
are interested enough in file forwarding to go to court about it.

> Netdata  is  a  royal pain,  it  required  a  lot  more code  than  CARD,
> especially if you  want to do things decently. Above  all it is extremely
> frustrating to have to spend  lots of programming time encoding mandatory
> fields which contain garbage, and in very unpleasant formats to boot. But
> you don't need  to know the number  of records in advance,  nor the exact
> LRECL  (except  for fixed  format  files):  the  LRECL  is only  used  to
> dimension a buffer. When decoding, it is very easy to ignore the crap you
> don't want  to hear  about, but  spanned records  are not  convenient and
> writing code that won't go into a loop  if the input file is invalid is a
> pain.

The thought just occurred to me that by the time MAILER gets it's
hands on a Netdata file that needs forwarding, it's already in
the spool as card images. It could just be read w/o interpreting
the Netdata and packaged into octet streams for safe transport.
That places the burden of decoding the data on the final
recipient, but one could make the case that it is always the
responsibility of the receiving system to deal with the format of
the data in the file and concentrate on having the clients deal
with the Netdata.

I have a set of C routines somewhere around here that generate
and decode Netdata that were written as part of a do-it-yourself
BITnet connection for an obscure non-IBM machine. I don't know
how well they'd fit into MAILER, but John's welcome to them if he
wants them.
Comments: Gated by NETNEWS@AUVM.AMERICAN.EDU