Unify comment handling for all states in parsing lexer.