utilized tokenizer instead of splitting by white space