Refactor and test lex API. Simplify standalone use of lex module. Fix couple of bugs. Beautify test code.

### Lex:
* [will fill]

### Lex test:
* [will fill]
Check the tests added in 2a7ac8614415a2c938e50e50a1ddd93b76f97f23 and the commit message of 9ff9154fecd731e0fd7d549f9558534abe3773b3 .