Added convenience tokens generator.
Update Lex_tokens_ordering to use cleaner .tokens() interface.