zaach/jison — Issue #63: Tokenization error while its follow by alphabet.
Author: neizod
Created: 2011-10-14 21:11:31+00:00
State: open

**The Rule**

%lex
%%

"foo" return 'FOO';
[a-zA-Z] return 'TEXT';
<> return 'EOF';
" " /* do nothing */

/lex
%%

exp
: group EOF { return $1; }
;
group
: group e { $$ = $1 + $2; }
| e
;
e
: TEXT { $$ = yytext; }
| FOO { $$ = 'BAR'; }
;
So I try out:
`foofoo`
returns:
`fooBAR`
but i expected
`BARBAR`

Then I observe it by changing little rule:

[a-zA-Z&] return 'TEXT';

And try this:
`fooafoo`
return the wrong output:
`fooaBAR`. (expected `BARaBAR`)

While:
`foo&foo`
return the right output:
`BAR&BAR`.

Is there any way to tokenize text without relying on punctuation?