update test system to use external txt files
create separate tokens for leading whitespace
refactor: extract methods from tokenize_buffer