zaach/jison-lex — pull request #2 by GerHobbelt (2013-02-16 00:07:22+00:00)
Title: mutiple fixes and two features: comments=documentation in the generated lexer + 'backtracking' lexer power

# Management summary:

## Fixes:

- do the right thing when lexer action code contains multiple return statements: convert each token to its numeric value, not just the first occurrence.

- treat yylloc like you do yytext, yyleng et al: auto-expand to a `yy.` reference

- anything a little complicated in the lexer action code breaks the `Function()` construct (at least in Chrome/Chromium): fixed by providing an alternative way of constructing lexer action code from the user-defined %{...%} lexer action snippets.

- robust pushState()/popState() code

- fix inspired by code from another jison clone, which fixes the topState() error (wrong state returned)

## Features:

- add basic documentation to the lexer AND make sure that documentation ends up in the generated lexer as well; people who don't want that will/should run a minifier/obfuscator on the generated code anyway.

- options.backtrack_lexer: the lexer includes a mode to 'backtrack' i.e. try multiple matching regexes on an input each time a token is required: when the user invokes `lexer.reject();` in the %{...%} lexer action code, the current match is rejected and the subsequent regexes are tested in order to produce a desired token. If you don't use this feature, there is no extra cost.

The intent of the whole shebang is to produce a more powerful 'off the shelf' lexer which is easy/easier to debug (breaking statements across multiple lines helps debugging in Chrome a lot as you now actually see where you really are; the time of 80*25 VTs is far behind us so legibility criteria should adapt too).

(P.S.: I'm a fruitcake who pulls his generated parsers through a custom AWK script and jsbeautify to make sure the jison output is very legible and debuggable; that has saved quite some time on debugging the complex grammars. For production, one pulls the entire kaboodle through a minifier/obfuscator anyway, so I'd rather have my generated parsers be a bit verbose, particularly in the comment 'documentation' department.)

# The detailed change list

SHA-1: aa1c979
* fix: ensure that ALL tokens are converted to their numeric ID in the lexer actions. Of course the lexer MAY return strings but it is cleaner and faster to return numeric IDs every time.

This resolves the lexer output oddness for larger lexer action blocks where multiple 'return' statements can be placed in conditional sections inside the action block. for example (simplified):

```
([A-Za-z0-9_][A-Za-z0-9_]* %{
console.log("looking up row/column/named identifier token in symbol table: ", yytext, this, this.matches);
s = yytext.toUpperCase();
if (yy.symbolTable[s]) {
rv = yy.symbolTable[s];
yytext = {
text: yytext,
col: function() {
throw new Exception("I'll be buggered!");
}
};
return rv.token;
}

// Would it be a viable basic column identifier?
match = s.match(/^[A-Z]+$/);
if (match) {
// check if this is a legal column id:
return 'COLUMN_ID';
}
// Would it be a viable basic row identifier?
match = s.match(/^[0-9]+$/);
if (match) {
return 'ROW_ID';
}
return 'error';
%}
```

SHA-1: 640dd9b
* yylloc is now expanded by the lexer, just like yytext, yylineno and yyleng already were. (yylloc --> yy.yylloc)

SHA-1: 34b6474
* and the same 'barf hairball on good action code' problem in the lexer, fixed now. Unfortunately it also means that erroneous generated action code is only detected when you try to run the generated parser.
(I ran into the issue that jison is very whitespace sensitive: the lexer b0rked because a line separating two lexer rules contained a series of spaces and jison then decided that the second rule was simply some more action code of the first :-( )

SHA-1: 47206a3
* patched the code generator as it broke on the offending line (where the action code is poured into a Function()) for action blocks such as this one:

```
... rule ...
| TAN '(' e ')'
{
$$ = {
token: FKW_FUNCTION,
token_attr: FKA_ABSOLUTE,
childs: [null, $3],
loc_info: @$,
type: FT_NUMBER,
unit: $3.unit,
value: Math.abs($3.value)
};
}
```

The object creation like that would produce a 'Unexpected symbol' error.

SHA-1: cce66ae
* add some basic documentation to the generated parser for easier maintainability / usability of the generated code.

SHA-1: 872b540
- lexer.unput(): this.yyleng got out of sync with this.yytext.length as it was not updated after the location range has been updated using its previous value

SHA-1: 538e4fa
* lexer run-time: make sure that too many calls to popState() do not nuke the lexer condition stack.

SHA-1: a0e648e
* Merge remote-tracking branch 'remotes/atrniv/master': topState fix, done another way...

SHA-1: 08fee30
* Fixed an error in topState which used to return the penultimate state instead of the last state on the stack.

* removed unused member variable. (_less)

SHA-1: 5c92866
* added support for a 'backtracking' lexer: this is a standard (non-flex) lexer which for each matching regex will invoke the corresponding action code: when this returns a TOKEN (~ truthy value), then that token is produced (so far the behaviour is identical to the regular lexer); when no token is produced, then the context is rewound and the next matching regex is tried instead, until either a token has been produced or the lexer rules set (regex set) has been exhausted, in which case an ERROR exception is thrown (this last bit is identical to the default lexer too).

This behaviour can be controlled in the generated run-time by the options.backtrack_lexer boolean.

SHA-1: 34130aa
* The parser.lex() method has been corrected to correctly support the new lexer code, which will return boolean FALSE when no token was produced (so that .lexer.js() will recurse until a token has been produced by the lexer action code after all) -- I kept the original @zaach recursion in there instead of turning it into a loop as it would rather quickly cause a fatal error in the JavaScript engine when the lexer action code is broken (and never produces a token): replacing this recursion by a loop would hang the lexer more or less indefinitely, making debugging such a situation much harder.

* augmented the lexer condition state access methods to improve robustness (e.g. against overzealous use of popState() in user action code)

* topState() now can produce one of the pushed lexer condition states when a (positive, non-zero) numeric index is passed as a parameter: topState(0) is identical to topState() and produces the currently active lexer condition state, while topState(1) produces the previously active lexer condition state, etc.

SHA-1: 53b6a35
* lexer: corrected the backtracking feature: the user must call the lexer method .reject() to signal that the current lexer rule FAILED to match a suitable token. (This fixes the scenario where a lexer rule is used to skip whitespace: such a rule won't return a token, nor call more() and should still 'succeed'.)

* lexer: added the ._backtrack boolean member variable to track the invocation of reject() in the lexer action code.

SHA-1: 053f899
* corrected the backtrack_lexer option name in the run-time exception when reject() is invoked in the action code inside a non-backtracking lexer

* added lexer unit tests to verify proper backtrack_lexer operation