refactoring the tokenizer and preprocessor