added testing function for simple tokenizer