Improved support for zero-length lexer tokens

Fixes #687 
Fixes #688