Unit tests of LexerEngine cover test case up to TC52.

Expanded the unit tests of LexerEngine to cover test cases TC1..TC52.