moving tokenize_gen, jieba and nltk from vectorizer to tokenizer