SHA-1: aa1c9790a4576d20874c31339001c3b4b1becdc1

* fix: ensure that ALL tokens are converted to their numeric ID in the lexer actions. Of course the lexer MAY return strings but it is cleaner and faster to return numeric IDs every time.

This resolves the lexer output oddness for larger lexer action blocks where multiple 'return' statements can be placed in conditional sections inside the action block. for example (simplified):

([A-Za-z0-9_][A-Za-z0-9_]*         %{
    console.log("looking up row/column/named identifier token in symbol table: ", yytext, this, this.matches);
    s = yytext.toUpperCase();
    if (yy.symbolTable[s]) {
        rv = yy.symbolTable[s];
        yytext = {
            text: yytext,
            col: function() {
                throw new Exception("I'll be buggered!");
            }
        };
        return rv.token;
    }

    // Would it be a viable basic column identifier?
    match = s.match(/^[A-Z]+$/);
    if (match) {
        // check if this is a legal column id:
        return 'COLUMN_ID';
    }
    // Would it be a viable basic row identifier?
    match = s.match(/^[0-9]+$/);
    if (match) {
        return 'ROW_ID';
    }
    return 'error';
%}
gitignore bak files
SHA-1: 640dd9b77f5a1d20c24828209885dbd105a830f4

* yylloc is now expanded by the lexer, just like yytext, yylineo and yyleng already were.
SHA-1: 34b6474759541441c4355f13e4f8faa004204bbd

* and the same 'barf hairball on good action code' problem in the lexer, fixed now. Unfortunately it also means that erroneous generated action code is only detected when you try to run the generated parser.
(I ran into the issue that jison is very whitespace sensitive: the lexer b0rked because a line separating two lexer rules contained a series of spaces and jison then decided that the second rule was simply some more action code of the first :-( )

SHA-1: 47206a3350d3359a8190c6ff342c42a17bc53d55

* patched the code generator as it broke on the offending line (where the action code is poured into a Function()) for action blocks such as this one:

    ... rule ...
    | TAN '(' e ')'
        {
            $$ = {
                token:      FKW_FUNCTION,
                token_attr: FKA_ABSOLUTE,
                childs:     [null, $3],
                loc_info:   @$,
                type:       FT_NUMBER,
                unit:       $3.unit,
                value:      Math.abs($3.value)
            };
        }

The object creation like that would produce a 'Unexpected symbol' error.
SHA-1: cce66aee9582eb4e0faf051bd2cd19698cf9a676

* add some basic documentation to the generated parser for easier maintainability / usability of the generated code.
SHA-1: 872b540eccbc82c3367af543322ad1bf572a9a17

- lexer.unput(): this.yyleng got out of sync with this.yytext.length as it was not updated after the location range has been updated using its previous value
SHA-1: 538e4fa1b0352da73862d0985683beb3f70e2e6f

* lexer run-time: make sure that too many calls to popState() do not nuke the lexer condition stack.

SHA-1: a0e648ec027220ad6d0f85dbf5a6f9d38bc3e97c

* Merge remote-tracking branch 'remotes/atrniv/master': topState fix, done another way...

SHA-1: 08fee3031573417ade42a84205c6974f78c7f60b

* Fixed an error in topState which used to return the penultimate state instead of the last state on the stack.
removed unused member variable.
outlining (whitespace/braces) changes before merging in the lexer backtrack feature code.
SHA-1: 5c928663baf248e5e90ca55f5d09335b41fcf3b0

* added support for a 'backtracking' lexer: this is a standard (non-flex) lexer which for each matching regex will invoke the corresponding action code: when this returns a TOKEN (~ truthy value), then that token is produced (so far the behaviour is identical to the regular lexer); when no token is produced, then the context is rewound and the next matching regex is tried instead, until either a token has been produced or the lexer rules set (regex set) has been exhausted, in which case an ERROR exception is thrown (this last bit is identical to the default lexer too).

This behaviour can be controlled in the generated run-time by the options.backtrack_lexer boolean.

SHA-1: 34130aab1f9477180dbe058c7a03ecc5d1ad531b

- The parser.lex() method has been corrected to correctly support the new lexer code, which will return boolean FALSE when no token was produced (so that .lexer.js() will recurse until a token has been produced by the lexer action code after all) -- I kept the recursion in there instead of turning it into a loop as it would rather quickly cause a fatal error in the JavaScript engine when the lexer action code is broken (and never produces a token): replacing this recursion by a loop would hang the lexer more or less indefinitely, making debugging such a situation much harder.

- augmented the lexer condition state access methods to improve robustness (e.g. against overzealous use of popState() in user action code)

- topState() now can produce one of the pushed lexer condition states when a (positive, non-zero) numeric index is passed as a parameter: topState(0) is identical to topState() and produces the currently active lexer condition state, while topState(1) produces the previously active lexer condition state, etc.

SHA-1: 53b6a35d9a99db593e9d9a837573b847e650a26c

* - lexer: corrected the backtracking feature: the user must call the lexer method .reject() to signal that the current lexer rule FAILED to match a suitable token. (This fixes the scenario where a lexer rule is used to skip whitespace: such a rule won't return a token, nor call more() and should still 'succeed'.)

- lexer: added the ._backtrack boolean member variable to track the invocation of reject() in the lexer action code.

SHA-1: 053f89910c75a6dfd328b2c6f867bfe0cb27973f

* - corrected the backtrack_lexer option name in the run-time exception when reject() is invoked in the action code inside a non-backtracking lexer

- added lexer unit tests to verify proper backtrack_lexer operation
whitespace police: trim all WS; UNIX LF; tabs to spaces.
added lexer.stateStackSize() method: "return the number of states currently on the stack" -- useful when you want to know the state stack index, e.g. when you want to store attributes with each state (such as counters which help drive the lexer actions).
Merge pull request #2 from GerHobbelt/master

mutiple fixes and two features: comments=documentation in the generated lexer + 'backtracking' lexer power