started multitoken case
refactored tokenizer to hide represenation during tokenization