Refactor and test lex API. Simplify standalone use of lex module. Fix couple of bugs. Beautify test code.

### Lex:
* [will fill]

### Lex test:
* [will fill]
As a public method, this needs testing.