Global changes

ReCreate lexer as Token Input Stream.

Previous code didn't process very large files and fell with heap
overflowing. And generally it was very expensive on memory.