Improved tokenizer thanks to Chan

Tokenizer was also moved in its own source file: tokenizer.c