add support for cut_all and tokenize for search