removed tokenizer's dependency on the code buffer