use macro to avoid starting another lexer for most tokens

This removes the warning about the lexer accepting empty input and could
potentially be faster.
simplify handling of multiple tokens
Merge pull request #43 from kmicklas/master

Clean up lexer a bit