vastly improved tokenization, consistent use of tokenizer for all tools, updated tests accordingly