fixed tokenizing of dots and tabs
Revert "correct the code"

This reverts commit 606ccb8a689155708ffee08e94e184bfda97ac36.
Revert "add lexertest"

This reverts commit 3b55ddd96b5fd4e2cf056b27d39ce2c5b57c008a.