cooperative-computing-lab/cctools — pull request #556 by btovar (2014-09-20 01:23:03+00:00)
Title: Makeflow lexer and parser

Incorporates the new lexer and parser in makeflow. Of note:

- lexer_book is a structure that does all the book-keeping. lexer_book keeps a pointer to the environment in which variable expansions are done. The parser updates this environment (e.g., the value of node variables, and shell variables), but all the expansion are done by the lexer. The parser only sees literals and syntax tokens.

- The lexer has four kind of functions: low level to accumulate tokens, recursive functions to accumulate lists of tokens depending of the syntax (e.g. command or list of files tokens), recursive functions to expand variables, and functions to interface the parser.

- There is better error reporting. Line and column of the error is now reported.

- mflex and -mfpar are now available to debug the lexer and parser, respectively. Also, -d makeflow can be used to debug all makeflow operations (I have not updated makeflow.c to use this).

- All tokens are read on-demand (the parser calls lexer_next_token()). This should make easy to resurrect @pbui idea on reading a makeflow from stdin.

- The parser is much simpler, as it only needs to assemble the tokens in the correct way.

- Since the parser is simpler, it is easy to add lexer tokens to extend makeflow syntax, such as @pbui rule expansions #211 .

- Similarly, it is easy to add other types of variable definitions (i.e., in addition to = and +=). For example, we could add var != shell command, and restrict it to never appear after the first rule has been defined.

CHANGES THAT BREAK COMPATIBILITY:

- Variables are strict now, in the sense that the workflow will fail if a variable has not been defined by the time of the expansion (including environment variables).

- A command line should always start with \t. Before any number of spaces was allowed.

- The syntax \t@var=... is not allowed now. After a file specification, the only line that can start with \t is a command line.

- Syntax @var=... is deprecated. The @ is not needed since the context of the variable is clear following file specification.

- Variable expansions can be written as $(VAR), or ${VAR}, or simply $VAR if followed by any non alphanumeric character.

- Single and double quotes follow standard definitions (no expansion in single quotes, no lines are broken inside quotes, variable expansion is allowed inside double quotes, \n, \t are expanded inside double quotes, etc.).

- In VAR=\$VAR1, the value of $VAR1 is only expanded if used in a command line (the \ tells the lexer to ignore the $, and the expansion is done by the shell when executing the command). This means that $VAR1 should be an exported variable. Before, the same effect was achieve by VAR=$$VAR1. Now, $$ triggers an error, as $ is not a valid variable name character. The new syntax should communicate the intent better.

These changes follow the proposed makeflow specification at the internal CCL memo (see webpage).