Working tokenizer implementation.