fixed tokenizer; includes new tests