From: anders@lise3.lise.unit.no (Anders Christensen)
Subject: Eric's question
Organization: /home/flipper/anders/.organization
Date: 29 Aug 92 05:05:37

Eric Thomas <eric@sejnet.sunet.se> wrote:

> Ok, time for the usual stupid question. Say I have a program that does:
> 
>                   Parse var line ':'tagname'.'value' :'line

The short answer: 

I don't think you need to do very much. The best is just to let it be,
and stick to a strict definition of NAMES files that requires that
there is a Space before the colon. Or you can redefine the
definition of the NAMES file format; you can make the parse clause
more powerful, in order to handle the situation; or you can write more
clauses to handle the situation without assuming anything about the
underlaying operating system.

The long answer:

I assume:

   * You have a program containing this line, and you have a NAMES file
     as data which it shall process.

   * You want to be able to use the program on several platforms,
     let's use two platforms for the sake of the discussion: a CMS
     machine using EBCDIC and a single Space character as one
     platform, and a Unix machine using ASCII and interpreting both
     Tab and Space as whitespace. These seems to be the platforms in
     question in this discussion. 

   * Neither the data, nor the program it self is networked. You use
     ftp in text mode to move the files between the machines, but you
     do not want to do manual editing on the files after the ftp. 

   * In particular, you want to be able to fetch a NAMES file from one
     machine to another machine without changing it, and you want to
     be able to use the same rexx script code on both machines,
     without any editing. 

Am I right so far?

1) Since NAMES files are a CMS concept, let's start on the CMS machine. 
   It only recognizes Space as whitespace, the NAMES file will contain
   the Space character in front of the colon, and the parse will work
   just fine. 

2) You ftp both the NAMES file and the rexx script to a unix machine
   and want to run it there. During the ftp process, the conversion
   from EBCDIC to ASCII has translated the EBCDIC Space characters to
   ASCII Space characters. Consequently, both the rexx script and the
   NAMES file contains the Space character, and it will work. 

3) You edit the NAMES file with a Unix editor, which inserts a tab
   instead of the space in front of the second colon. First let's see
   what happens when you ftp the file back to CMS. During the ftp
   process the ASCII characters are translated to EBCDIC. Since EBCDIC
   don't use Tabs, the Tabs in ASCII are translated into EBCDIC
   Spaces. When running it on the CMS machine, there will not be any
   problems. 

4) Then back on the Unix machine, you want to run the program with the
   NAMES file containing a Tab in front of the second colon, which
   will not work, since the rexx script requires it to be a Space. 
   
The real problem is that the standard of NAMES files _requires_ the
character in front of the second colon to be a Space (correct me if I
am wrong). At least, that is how your program interprets it; a Space
followed by a colon is a new tag/value; while a colon preceded by
anything else then an Space (Tabs included) is part of the preceding
'value'.

If that is your definition of the format of a NAMES file, then you
must make sure that on systems using Tabs, you set your editor in mode
that don't generate Tabs, that you filter away the Tabs, or that you
in other ways don't insert a Tab there. Consequently, the NAMES file
format is not very portable, since it relies on CMS assumption of only
one whitespace character.

Let us then assume that you redefine the NAMES file format, so that it
requires the character in front of the second colon to be 'blank' or
whitespace, i.e. not a particular character, but any in a set of
whitespace characters. (Note: I have no wish or intention to redefine
the format of NAMES files under CMS, but Eric's suggestion indicated
that a problem would rise when a non-Space whitespace character was
used in front of the second colon. Then the format of NAMES files
would have been extended to allow any whitespace character there ...
else Eric's NAMES file would be incorrect.) 

Under this assumption, the parse clause becomes unportable. It assumes
that Space is the only whitespace character. It will still work under
CMS, since this assumption hold for that environment. In the first
three scenarios listed above, there will be no problem. In scenario
two, the program contains a bug, although it is not triggered.

Scenario four poses a problem, since the bug in the parse statement is
triggered. The real problem here, is that the parse clause is not
powerful enough to handle the situation that occurs when a pattern
(either constant or variable) is supposed to match a non-static
string. 

Then, what can be done (except from just accepting that the format of
NAMES files requires a Space):

 1) You could redefine the NAMES file format, so that it had a syntax
    under which the delimiters (now some whitespace + colon) were
    constant. That would mend the problem with the parse clause not
    being able to match varying strings (Note: I do not refer the
    "variable patterns" as opposed to "literal patterns", I refer to a
    pattern (literal or variable) matching not _one_ specific sequence
    of characters, but several alternative sequences of characters).

    I suppose that is not really attractive, since you don't want to
    put down a lot of work into porting the system to UNIX. But _if_
    portability is your prime goal, then you should not base yourself
    on the concepts of one operating system.

 2) You could do it the other way. Redefining the parse clause to be
    able to match varying patterns. Then you are very close to what is
    called "regular expressions" in other operating systems (more
    about that later). This is perhaps not so attractive, since you
    may have lots of code using this construct, and even _if_ such a
    construct is included in the standard, there will be a period of
    overlap where old systems are not compatible with new systems.

 3) You could rewrite the code, either to be a larger structure (as
    suggested by Scott Ophof), or you could use functions instead of
    the parse clause. This is more or a mixture of the previous two,
    and is not very elegant, but it will work with today's syntax and 
    on the NAMES files containing Tabs.

So, to what extent is this a problem? I will assert that most Rexx
code on a CMS machine will solve problems that do not exist on e.g. a
Unix machine (I will make an exception for XEDIT macros, iff you have
an XEDIT-compatible editor running on Unix). Most, if not all CMS Rexx
scripts that use commands, will need heavy porting work to run under
Unix. And for most of these, it is easier to rewrite the code than to
try to port it. 

Portability comes in two flavors: Portability of the scripts, which is
virtually non-existing between CMS and Unix, due the large differences;
and portability of the language, i.e. the users' knowledge of the
programming language Rexx. Even though the first is sometimes almost
non-existing, the second is very valuable, since it allows the user to
bring his or her knowledge to another platform. 
 
Then, about regular expression. In the parse statement you seek for a
Space followed by a colon. That is a constant string of length two
characters. Suppose you want to seek for either a Tab followed by a
colon or a Space followed by a colon. There is no way in the current
standard of Rexx to do that in one parse (at least very hard). This
situation is just a special case. Suppose that you wanted a pattern in
the parse clause to match either an 'a' before a colon or a 'b' before
a colon? There are no (easy) way to do that in Rexx either. Or suppose
that you wanted the pattern to match any number of 'a's before two
'b's? You can match a specific number of 'a's before two 'b's, but not
an arbitrary number of 'a's.

How is this done with regular expressions? I'll show it, using Unix
syntax below (Any flames that don't add anything constructive to this
discussion, please send them directly to me by email, not to the net),
using each of the situations above as examples.

In both Rexx patterns and Unix regexp:

    "a:" matches an 'a' followed by a colon
    "b:" matches a 'b' followed by a colon

and in Unix regexp:

    "[ab]:" matches an 'a' or 'b' followed by a colon

In both Rexx patterns and Unix regexp:

    "aaaaabb" matches five 'a's followed by two 'b's

in Unix regexp:

    "a*bb" matches any number of a's followed by two 'b's.

How does this solves the problem of the Space or Tab before a colon?
Well, using Unix regexp, it could be coded as:

    "[ \t]:" matches Tab or Space before colon

Here, Tab is coded as a special character: 't' preceded by
backslash. In this case, you are not even dependent on the character
set specific notation of using a hexadecimal literal to represent a
character not allowed in the character set. I.e. the Rexx notation
'09'x matches Tab only on ASCII machines, while the Unix regexp
notation "\t" matches Tab on any system, independent of the character
set.

Now, this doesn't really solve the problem, since Tab is not a
whitespace character under CMS, even though it might be considered
such under Unix. The way this is solved in Unix regexp, is by using a
'character class', that is a special character which match any of a
certain set of (possibly system defined) characters. E.g. the special
character "\s" is sometimes used under Unix in regexps to match any
whitespace character, it is really just an expansion for "[ \t]" or
whatever is commonly thought of as whitespace. Under system having
only one whitespace character, it would match only that. The task of
matching a whitespace character in front of a colon then becomes:

    "\s:"

Regexps are very powerful, and definitively more powerful than the
pattern matching of parse. Including regexps into parse' pattern
matching would indeed add to its power. One example, how can parse
take a pattern that matches the syntax of a literal string? It can't,
you have to work around it, and use additional functions and parse
clauses. The following regexpr will match a number of characters
delimited by either single or double quotes, which may contain 
adjacent pairs of same quote within the pattern matched:

    ('([^']|'')*'|"([^"]|"")*")

And this shows the negative side of regexps too. When they get
complicated, they are not particular easy to read. So it is really
a trade-off between power and easiness. Getting the maximum of both is
really difficult, although Rexx as defined in TRL has managed to
balance this problem very good. I can understand people who want to
include regexp to get the added power, but I can also understand those
who don't want the (admittingly) hairy syntax into Rexx. Anything you
can do in Unix using regexps can be done in Rexx using parse and
functions, it just takes a few more clauses ... and sometimes it take
a lot more clauses. If you are not willing to add a new feature that
gives the language more power, then you should be prepared that you
have to write longer scripts.

That concludes my long answer to Eric's question. To summarize: The
clause you listed will be a problem in one case: when you move the
CMS-specific parse-clause to Unix, and expect to use the NAMES files
in "Unix-format". The parse clause you listed is not easily able to
handle the NAMES file format (if Tabs are allowed as a whitespace
character) under Unix, due to CMS specific assumptions.  Consequently,
you can either redefine the NAMES file format; extend Rexx with more
powerful features; or you can add more Rexx clauses to solve the
problem in a way that makes no assumptions about the underlaying
operating system; or you can simply require your script under Unix to
use the NAMES file format of CMS.

-anders
Newsgroups: comp.lang.rexx