Newsgroups: bit.listserv.lstsrv-l
Date:         Wed, 15 Apr 1992 00:16:47 +0200
From:         Eric Thomas <ERIC@SEARN.BITNET>
Subject:      Help needed for testing exotic forms of Netdata files

I  have just  finished debugging  generic input  spool file  manipulation
routines for release 1.7d (yes, d - no time to beta test this in time for
1.7c). These  routines will make  it possible  to use formats  other than
PUNCH with acceptable  performance, and without having to  stage the data
to a  disk file  and re-read  it. They  will also  simplify the  logic of
routines which  deal with incoming files,  as all the kludges  for things
like PROFS or MVS can be put in a single place and then forgotten about.

Now, of  course, there is  this one problem  that the description  of the
Netdata  format is  at  best unclear.  Oh,  the way  data  is split  into
segments of up to 253 bytes so we  can save one byte per segment for mail
files and lose a bunch of pairs of bytes (and all the cycles they take to
be reassembled) for binary files is  pretty clear. The way text units are
assembled into control records is easy to understand. What the text units
mean,  however, is  not  always  clear, and  there  isn't  a single  line
describing  what the  data records  contain, and  under what  conditions.
Looking  at source  code makes  it even  less clear  what really  belongs
where,  it looks  like the  decoding process  is a  pretty happy-go-lucky
operation - we try  not to worry too much about what  the final file will
look like, it's probably fixable  through some FILEDEF option or filemode
A4 anyway.

Supporting VM files  is easy - one  data record is one  input record from
the original file,  no prefix, no adjustment, no problem.  Same thing for
JNET. There seems to be a potential  however for MVS files where one data
record would be one disk block, or  a record count followed by the actual
data, or  many other 2-3 letter  sequences whose meaning I  can guess but
which  I have  never actually  used. I  have absolutely  no intention  of
supporting anything that native VM commands do not handle properly, but I
would like to make sure I remain  compatible with 1.7c. In addition, I am
not sure how my code will react to  some types of files which I have seen
described in  the sources  but never  actually seen. So  I would  like to
solicit  help from  MVS sites,  especially  those without  a decent  mail
system ;-),  and would  like to  receive sample  files in  Netdata format
exhibiting the following arcane properties:

- Two files in the deck, but one is a note so VM can process it.

- More than one file in the deck, in a combination VM won't handle.

- Some dsname with at least 3 components, like AA.BB.CC.DD.

- Some dsname of the type AA.XYZ.OUTPUT but with a INMTERM unit.

- Any record format you can think of which it is reasonable for end users
  to try to send  to LISTSERV. This may or may not include  U, FB, VB and
  VBS - I just don't know what normal editor sessions come up with.

- Your favourite mail system, as long as it is supported by 1.7b.

Finally, here are some  crude benchmarks I did when I  ran the program on
large  files  to make  sure  it  produced  correct output.  The  routines
basically extract and return strings mapping a logical record, one record
at a time, so  I wrote a small program that extracts  a record and writes
it to  a disk file until  EOF, to compare  with the original and  see how
long it takes.  DISK format is particularly slow because  it requires two
passes on the reader file - the LRECL  is in the last card, and you can't
return the first record until you know  it (DISK LOAD, on the other hand,
only  needs  to  write the  raw  data  to  disk  and sort  out  the  mess
afterwards). CARD was added for the  sake of completeness, because it was
easy to  do and  could be useful  to have, and  wasn't optimized  at all.
Netdata is reasonably optimal on the other hand, I guess as optimal as it
can get in  PASCAL. As expected, CARD  is the fastest and  the native CMS
commands are always  slower, but this wasn't the case  until I remembered
to set up my  data buffers to allocate in blocks of 8k  and not 128 bytes
:-)

  Eric

    +----------------+----------------+---------------+
    | LSWPLIB MODULE | Native command |  LSWISF+FIO   |
    | (6 recs, 220k) | VCPU TCPU I/O  | VCPU TCPU I/O |
    +----------------+----------------+---------------+
    | DISK LOAD      | 0.55/1.33  36  | 0.23/0.67  31 |
    +----------------+----------------+---------------+
    | CARD LOAD      | 0.12/0.30  33  | 0.24/0.43  31 |
    +----------------+----------------+---------------+
    | DMSDDL RECEIVE | 0.34/0.87  12  | 0.13/0.30  31 |
    +----------------+----------------+---------------+

    +----------------+----------------+---------------+
    | OSVSAM MACLIB  | Native command |  LSWISF+FIO   |
    | (968kb)        | VCPU TCPU I/O  | VCPU TCPU I/O |
    +----------------+----------------+---------------+
    | READCARD       | 1.38/3.71 128  | 0.33/0.93  35 |
    +----------------+----------------+---------------+
    | DISK LOAD      | 2.27/5.65  89  | 1.20/2.90  35 |
    +----------------+----------------+---------------+
    | CARD LOAD      | 0.51/1.06 128  | 1.52/1.92  35 |
    +----------------+----------------+---------------+
    | DMSDDL RECEIVE | 2.47/4.87 124  | 0.85/1.45  35 |
    +----------------+----------------+---------------+
Comments: Gated by NETNEWS@AUVM.AMERICAN.EDU