Added Public Interface to tokenizer. Tokenizer now complete enough to test