enable Tokenizer to handle a directory