changed tokenizer to be agnostic of complex syntax