change tokenizer to accept a returned character array.
deprecate, but not remove yet.