Package: clean-text
clean-textUser-generated content on the Web and in social media is often dirty. Preprocess your scraped data withclean-textto create a normalized text representation. For instance, turn this corrupted input:A bunch of \\u2018new\\u2019 references, including [Moana](https://en.wikipedia.org/wiki/Moana_%282016_film%29).


»Yóù àré     rïght &lt;3!«into this clean output:A bunch of 'new' references, including [moana](<URL>).

"you are right <3!"clean-textusesftfy,unidecodeand numerous hand-crafted rules, i.e., RegEx.InstallationTo install the GPL-licensed packageunidecodealongside:pipinstallclean-text[gpl]You may want to abstain from GPL:pipinstallclean-textNB: This package is namedclean-textand notcleantext.Ifunidecodeis not available,clean-textwill resort to Python'sunicodedata.normalizefortransliteration.
Transliteration to closest ASCII symbols involes manually mappings, i.e.,êtoe.unidecode's mapping is superiour but unicodedata's are sufficent.
However, you may want to disable this feature altogether depending on your data and use case.To make it clear: There areinconsistenciesbetween processing text with or withoutunidecode.Usagefromcleantextimportcleanclean("some input",fix_unicode=True,# fix various unicode errorsto_ascii=True,# transliterate to closest ASCII representationlower=True,# lowercase textno_line_breaks=False,# fully strip line breaks as opposed to only normalizing themno_urls=False,# replace all URLs with a special tokenno_emails=False,# replace all email addresses with a special tokenno_phone_numbers=False,# replace all phone numbers with a special tokenno_numbers=False,# replace all numbers with a special tokenno_digits=False,# replace all digits with a special tokenno_currency_symbols=False,# replace all currency symbols with a special tokenno_punct=False,# remove punctuationsreplace_with_punct="",# instead of removing punctuations you may replace themreplace_with_url="<URL>",replace_with_email="<EMAIL>",replace_with_phone_number="<PHONE>",replace_with_number="<NUMBER>",replace_with_digit="0",replace_with_currency_symbol="<CUR>",lang="en"# set to 'de' for German special handling)Carefully choose the arguments that fit your task. The default parameters are listed above.You may also only use specific functions for cleaning. For this, take a look at thesource code.Supported languagesSo far, only English and German are fully supported.
It should work for the majority of western languages.
If you need some special handling for your language, feel free to contribute. 🙃Usingclean-textwithscikit-learnThere is alsoscikit-learncompatible API to use in your pipelines.
All of the parameters above work here as well.pipinstallclean-text[gpl,sklearn]pipinstallclean-text[sklearn]fromcleantext.sklearnimportCleanTransformercleaner=CleanTransformer(no_punct=False,lower=False)cleaner.transform(['Happily clean your text!','Another Input'])DevelopmentUse poetry.ContributingIf you have aquestion, found abugor want to propose a newfeature, have a look at theissues page.Pull requestsare especially welcomed when they fix bugs or improve the code quality.If you don't like the output ofclean-text, consider adding atestwith your specific input and desired output.Related WorkGeneric text cleaning packageshttps://github.com/pudo/normalityhttps://github.com/davidmogar/cuccohttps://github.com/lyeoni/prenlphttps://github.com/s/preprocessorhttps://github.com/artefactory/NLPretexthttps://github.com/cbaziotis/ekphrasisFull-blown NLP libraries with some text cleaninghttps://github.com/chartbeat-labs/textacyhttps://github.com/jbesomi/textheroRemove or replace stringshttps://github.com/vi3k6i5/flashtexthttps://github.com/ddelange/retrieDetect dateshttps://github.com/scrapinghub/dateparserClean massive Common Crawl datahttps://github.com/facebookresearch/cc_netAcknowledgementsBuilt upon the work byBurton DeWildeforTextacy.LicenseApache