simplified applying the tokenizers.
slightly simplified lexer rules.