Refactor and test lex API. Simplify standalone use of lex module. Fix couple of bugs. Beautify test code.

### Lex:
* [will fill]

### Lex test:
* [will fill]
Why isn't this the default behavior of `tokenize` when `logger` is set to `None`?