Improved tokenizer for new language constructs and adaptive parsing