From: KPETERSO@BENTLEY.BITNET
Newsgroups: comp.os.vms
Subject: Summary of "Editing Large Files With TPU" Responses
Date: 11 Sep 92 00:18:00 GMT
Organization: The Internet

Again, thank you to everyone who responded to my query about eve-ing a
large file.  Enclosed is the original question, a summary of responses, and
what I did.  I have summarized as much as possible without losing the gist.
 _______________
|         ____  |  ---------------------------------------------------
|  ___   /    \ |   Karen Peterson           BITNET:  KPETERSO@BENTLEY
| /   \  |    | |   Bentley College          Phone:   (617) 891-3432
| |   |  |    | |   Waltham, MA  02154
| -----  ------ |
|_______________|  ---------------------------------------------------

*************** Original Message:  ***************
I am trying to EDIT/TPU (EVE) a 29553 block file and coming up with errors.
When I EVE the filename, the machine thinks about it, then I recieve the
following error message:  "2098 byte record to large for users buffer"
[...] UAF quota stuff deleted
We are running on a 6520 with 128 M memory, VMS V5.42, and disk quotas are not
a problem. Please respond to me directly at the address below.

******************** Responses: ********************
From: Mighty Firebreather <dragon@nscvax.princeton.edu>
IN%"dragon@nscvax.princeton.edu"  "Mighty Firebreather" 26-AUG-1992 14:41:51.62

        The problem is not your quotas or the memory available on your
machine.  The problem is that EVE, like most text editors, was designed to
edit *text* and the authors assumed a reasonable record length.  The limit
is something like five or six hundred characters per line (record).

If you must edit this file, you could try TECO or EMACS.

********************
IN%"DWING@UH01.Colorado.EDU"  "Dan Wing" 26-AUG-1992 14:47:01.06
From: Dan Wing <DWING@UH01.Colorado.EDU>

        I'll bet the problem is a long record in the file, which is longer
than EVE can handle.
        What is the facility that is reporting the error?  Is the error
prefixed with %SYSTEM, or %TPU, or ??
        Something not related to your problem, but something to check:  ensure
that the SYSGEN value for CHANNELCNT is larger than 200 (FILLM).

********************
IN%"WHITFILL@meediv.lanl.gov" 26-AUG-1992 15:41:00.59
From: WHITFILL@meediv.lanl.gov (Jim Whitfill - Los Alamos)

The problem is not the file size, but the record size. The file doesn't have
any cr/lf in it maybe? Do a dir/full on the file and look at the record
structure.

********************
IN%"FAIRFIELD@SCS.SLAC.STANFORD.EDU"  "Ken Fairfield; [415] 926-2924 SLAC" 26-AU
   G-1992 17:15:47.54
From: "Ken Fairfield; [415] 926-2924 SLAC" <FAIRFIELD@SCS.SLAC.STANFORD.EDU>

    The problem is NOT with your process quotas, it is with the FILE you are
trying to edit.  TPU can ony handle "lines" of something like 960 characters
in length (look in Chapter 2 of the VAXTPU manual, p.2-22/23 of the V5.2
docs that I have).  You _will_ get the first 960 characters of the long
record, but not the others.  What's worse, if you save a new version of the
file (ctrl/z out of EVE), that long record will have been _truncated_ back
to the first 960 characters, probably destroying the integerity of the file
vis-a-vis whatever application next wants to read it...

    So the question is, why does the file have such long lines?  (Note that
even the usual binary files under VMS, like .EXE, .OBJ, .OLB, are fixed
length, 512 byte records.)  The two common circumstances that I am familiar
with are (a) someone decides to write, e.g. from a Fortran program, to a
file with a maximum record length of 32765 bytes, or some such thing, or (b)
the file is written as Stream_LF, nominally to be more compatible with Unix
and C.  In such a case, e.g. with an image file that's to be sent to a laser
printer, there may not happen to be an LF character for a LONG stretch.
This latter case is common when FTPing a file from another non-VMS site...
but there are work-arounds...
    I might also point you to the TPU news group, VMSnet.tpu, which is
gatewayed to the mailing list INFO-TPU.  If your news system doesn't get the
VMSnet hierarchy, you can sign on to INFO-TPU by sending a one-line mail
message to:    LISTSERV@SHSU.BITNET
with the contents:    SUBSCRIBE INFO-TPU your_preferred_name

********************
IN%"BEAR@NUC.anu.edu.au" 26-AUG-1992 17:18:41.89
From: BEAR@NUC.anu.edu.au
Reply-To: BEAR@NUC.ANU.EDU.AU

[...]   it is the record-size that is too big - your version of TPU cannot
edit a file with line lengths of 2098 characters. I don't know what the limit
is on the TPU version you a running; on my system (V5.5), I could easily edit
a 2098 byte record (just tried it now), but it would be in the manuals.
        ... it is neither disk space nor memory that causes the trouble - and
whatever your limit is, it's gotta be better than EDT's a pathetic 255
character limit!
        The work-around is to write some small program in your favourite
programming language (DCL excluded!) that will chop the file up (search for
<CR> or <LF> characters and so on, or use brute force:

        program CHOP

c       Program to chop up a file with 2098 byte records (max).
c       Do a DIR/FULL OLDFILE.DAT (or whatever you call it) to
c       find the maximum record length first.

        character*2098  sbuffer
        byte            bbuffer(1:2098)
        equivalence     (sbuffer(1:1),bbuffer(1))

        open(unit=1,file='oldfile.dat',status='old',readonly,err=70)
        open(unit=2,file='newfile.dat',status='new',carriagecontrol='list')

10      continue
        read(1,20,end=30,err=40)lbuf,sbuffer
20      format(q,a)
        write(2,50,err=60)(bbuffer(i),i=1,lbuf)
50      format(132a1:)
        go to 10

30      stop 'Program ended'
40      stop 'Error reading old file'
60      stop 'Error writing new file'
70      stop 'Error opening old file'
        end

        that is the brute force method in FORTRAN It will give you max 132
characters per line (change to 80 if you must!).

********************
IN%"STUHART@EKU.BITNET"  "Joshua Hart -- A Forensic Chemist (to be) At Large!!!"
    26-AUG-1992 20:58:49.21
"Joshua Hart -- A Forensic Chemist (to be) At Large!!! <STUHART@EKU.BITNET>

        The maximum record length for a "record" (line) on the vax is 255 (255
characters). When the editor is processing the file it is finding records
which are in excess of that "quota" (for lack of a better word).. :)

********************
IN%"<@seq.uncwil.edu:jerry@marin.uucp>"  "jerry"
jerry <@seq.uncwil.edu:jerry@marin.uucp>
jerry%marin.uucp@seq.uncwil.edu

TPU can't handle records that large.  If this is a text file, it's probably
corrupt.  You'll find that EDT can't do it, either.  HOWEVER, you can dust
off TECO and use that, should the file not prove to be corrupt.  TECO treats
the file as a byte stream and doesn't care about record sizes.

********************
IN%"KARSTEN@tfl.dk"  "Karsten Nyblad, TFL, Denmark" 27-AUG-1992 03:48:19.39
"Karsten Nyblad, TFL, Denmark" <KARSTEN@tfl.dk>

My guess is that it is not an error!  I think your file contains a record of
length 2098 bytes, and TPU is not designed for handling that long records.
Try checking that with a DIR/FULL on the file.  The standard RMS I/O routines
keep track on the length of the longest record in a file, and that length is
listed as the maximum record length.  If the file has been written by VAX C,
you might not get the maximum record length.  Then you will have to copy it by
some program, perhaps copy or convert will use the standard RMS I/O routines.

********************
IN%"David.Hazledine@EMBL-Heidelberg.DE"  "David Hazledine"
David Hazledine <David.Hazledine@EMBL-Heidelberg.DE>

Do a DIRECTORY/FULL on the offending file and check the record format/record
attributes information.  You may well find the file has an unusual format and/or
very long record length which TPU is having problems dealing with.

********************
IN%"MEHTA@bms.com" 27-AUG-1992 12:40:46.84
From: MEHTA@bms.com

    I'm not totally sure if the following is true but I seem to remember
someone telling me that TPU downloads a whole file into memory when it is
envoked. It seems that you've run into this problem. I was also told that EDT
does not load into memory. Thus you may be able to do your edit in EDT if all
else fails.

********************
From: ptavv.llnl.gov!oberman@LLL-WINKEN.LLNL.GOV
Internet: koberman@llnl.gov

        Neither the size of the file, quotas, or the system are involved.
        TPU has an internal record buffer of fixed size. I should remember
what it is, but I don't right now. Either 1024 or 2048. I suspect the latter.
        Your file has at least one record of 2098 bytes. That blows TPU out of
the water. There is no way to edit such a file with TPU as far as I know. SED
should work on it, though. (No, I don't remember where to get it, either.
(Senility!)
        Since 2098 is a bit long for a normal text file, maybe it is corrupt,
or maybe you can use FILE or CONVERT to make it more manageable. For things
like PostScript files, FILE/RECORD_SIZE=128/TYPE=FIXED is pretty handy. But
don't try this on a variable record file. You may get total hash. I'm assuming
that the file is a stream type or a fixed type, already.

********************
From: Lorne Epp <van-bc!ubc-cs!mala.bc.ca!epp@UCBVAX.BERKELEY.EDU>
epp@mala.bc.ca

This means that the file contains a record that is too large for EVE to
handle.  Usually, this means that the file is not a text file.

********************
(michael@io.as.utexas.edu or UTSPAN::UTADNX::IO::MICHAEL [SPAN])

In article <1992Aug26.172043.1@ptavv.llnl.gov> oberman@ptavv.llnl.gov writes:
>Neither the size of the file, quotas, or the system are involved.
[...]
>but I don't right now. Either 1024 or 2048. I suspect the latter.

960.  Don't know why.

>Your file has at least one record of 2098 bytes. That blows TPU out of the
[...]

If you have the old DEC/Shell you got sed:  Do:
sed :== $sys$sysroot:[shellexe]sed
        but there are others around.  Or use TECO:  TECO:==$TECO32 TECO
Comes with VMS ;-)

********************
IN%"INFO-VAX@SRI.COM" 28-AUG-1992 12:19:59.48
From: Seth Grimes <dcatlas!sethg@UUNET.UU.NET>

Wasn't the record-length limitation to 960 characters lifted (for all
practical purposes -- new limit is large) as of VMS version 5.5?
********************

Dr. Kenneth H. Fairfield    |  Internet: Fairfield@Slacvx.Slac.Stanford.Edu
DECnet:   45537::FAIRFIELD (45537=SLACVX)   BITNET    Fairfield@Slacvx

In article <1992Aug27.115111.23317@dcatlas.dot.gov>, sethg@dcatlas.dot.gov (Seth
 Grimes) writes:
> Wasn't the record-length limitation to 960 characters lifted (for all
> practical purposes -- new limit is large) as of VMS version 5.5?

    I don't _think_ so...there was nothing in the V5.5, nor the V5.5-1,
release notes that refers to lifting this restriction...

********************
From: "FRED W. BACH"
Fred W. Bach ,    Operations Group        |  Internet: music@erich.triumf.ca

In article <0095FAD1.06919200.10772@nscvax.princeton.edu>,
 dragon@NSCVAX.PRINCETON.EDU (Mighty Firebreather) writes...
>
[...]
>
    Or, you could do what I do here.  This often happens and I come to the
rescue by KI-RESEARCH DEKnetting it over to a DATA GENERAL MV7800-C and then
I run SPEED on it.  (SPEED is a character editor which greatly resembles TECO,
and it has some nicer features).  Then I DEKnet it back.

********************
From: Blair Phillips - Digital
Blair Phillips                                  Blair.Phillips@cao.mts.dec.com

In article <1992Aug27.115111.23317@dcatlas.dot.gov>, sethg@dcatlas.dot.gov (Seth
 Grimes) writes:
>
>Wasn't the record-length limitation to 960 characters lifted (for all
>practical purposes -- new limit is large) as of VMS version 5.5?
>
        I have just created and editted a file with a 32767 character record
on VMS v5.5-2.
        Worked with LSEdit V4.0 too.

********************
From: Bernhard Fabricius
InterNet: BEAR@NUC.ANU.EDU.AU
    POSTMASTER@NUC.ANU.EDU.AU

In article <1992Aug27.104912.1@slacvx.slac.stanford.edu>,
                fairfield@slacvx.slac.stanford.edu writes:
|>In article <1992Aug27.115111.23317@dcatlas.dot.gov>, sethg@dcatlas.dot.gov
|>(Seth Grimes) writes:
|>> Wasn't the record-length limitation to 960 characters lifted (for all
|>> practical purposes -- new limit is large) as of VMS version 5.5?
|>    I don't _think_ so...there was nothing in the V5.5, nor the V5.5-1,
|>release notes that refers to lifting this restriction...

Furtunately, things don't _have_ to be listed in the release notes to be true.
I have quite happily edited 2098 character lines in TPU under VMS V5.5 (in fact
I am writing this with TPU/EDT emulation and [PF1][2/@][0/)][9/(][8/*][x] makes
a line with 2098 x's without problems!). So Karen can either upgrade to V5.5 or
use a small program to chop up the file in shorter bits if shw must edit it.

********************
From: netnews.louisville.edu!wkuvx1!goathunter@G.MS.UKY.EDU
goathunter@WKUVX1.BITNET, 502-745-5251

In article <1992Aug26.172043.1@ptavv.llnl.gov>, oberman@ptavv.llnl.gov writes:
[...]
> Your file has at least one record of 2098 bytes. That blows TPU out of the
> water. There is no way to edit such a file with TPU as far as I know. SED
> should work on it, though. (No, I don't remember where to get it, either.
> (Senility!)
Why, from my FILESERV, of course!  You can get my port of GNU sed via
anonymous ftp from ftp.spc.edu ([192.107.46.27]); you'll need:

        [.MACRO32]AAAREADME.TXT
        [.MACRO32]LZDCMP.EXE
        [.MACRO32.SAVESETS]SED.BCK_Z

Via e-mail, you can get it by sending the command SEND SED in the body
of a mail message to FILESERV@WKUVX1.BITNET.

********************
From: "David P. Murphy"
murphy@npri6.npri.com 602 Cameron St. Alexandria, VA 22314 (703) 683-9090

>> Wasn't the record-length limitation to 960 characters lifted (for all
>> practical purposes -- new limit is large) as of VMS version 5.5?
>>
>>sethg@dcatlas.dot.gov (Seth Grimes)
 >I don't _think_ so...there was nothing in the V5.5, nor the V5.5-1,
 >release notes that refers to lifting this restriction...
 >fairfield@slacvx.slac.stanford.edu (Dr. Kenneth H. Fairfield)
i just read, wrote, and just generally edited a file with lines up to
4000 bytes long in version V02-006 of TPU, linked 12-OCT-1991.
i am running VMS V5.5.

********************
From: "Frank Monroe, Jr."   InterNet  VAXS06@V7.VITRO.COM

In article <1992Aug27.115111.23317@dcatlas.dot.gov>, sethg@dcatlas.dot.gov (Seth
 Grimes) writes:
> Wasn't the record-length limitation to 960 characters lifted (for all
> practical purposes -- new limit is large) as of VMS version 5.5?
Yes.  I am using VMS 5.5-1 and TPU doesn't complain until I try to go beyond
32767!!

******************** What I Did ********************
What I ended up doing was a "vanilla" convert on the file (CONVERT input_spec
output_spec), then EVE-ing the output-spec (this was almost too simple!).  As
it turns out, the output_spec after the convert was over 8,000,000 lines long!.