use macro to avoid starting another lexer for most tokens

This removes the warning about the lexer accepting empty input and could
potentially be faster.
simplify handling of multiple tokens