Added tests for method tokenizer