more refactoring of tokenizers; tests for lextoken