Rewriting tokenizer function to be more generic
Adding debug code
Revert "Adding debug code"

This reverts commit eee4b047fbceb4b4eed570b7ca3a7b18e66c06f3.
Fixing the new stuff - now we have to adjust the tests & the AST processing