Revert "Revert "lexical analysis""

This reverts commit 5853589882f7dc15999e0a3194642b4d14209ad8.