improved docs and examples
refactored lexer state functions