working tokenizer wiith unit tests