Fixed tokenizer lines count

Signed-off-by: Grégory PLANCHAT <g.planchat@gmail.com>
Refactored TokenizerNavigationAwareTrait

Signed-off-by: Grégory PLANCHAT <g.planchat@gmail.com>
Fixed tokenizer lines count, added newline token

Signed-off-by: Grégory PLANCHAT <g.planchat@gmail.com>
Refactored TokenizerNavigationAwareTrait

Signed-off-by: Grégory PLANCHAT <g.planchat@gmail.com>
Fixed tokenizer lines count

Signed-off-by: Grégory PLANCHAT <g.planchat@gmail.com>
Updated lexer rules API, fixed semicolon search

Signed-off-by: Grégory PLANCHAT <g.planchat@gmail.com>
Updated lexer Constructor rule

Signed-off-by: Grégory PLANCHAT <g.planchat@gmail.com>
Fixed tokenizer regex parsing and lexer comment skipping

Signed-off-by: Grégory PLANCHAT <g.planchat@gmail.com>