utoken (pypi): utoken is a universal tokenizer (multilingual word segmenter) that divides text into words, punctuation and special tokens such as numbers, URLs, XML tags, email-addresses and hashtags. It comes with a companion detokenizer.
Keywords: machine|translation|datasets|NLP|natural|language|processing|computational|linguistics