Package: profanity-filter
profanity-filter: A Python library for detecting and filtering profanityTable of contentsprofanity-filter: A Python library for detecting and filtering profanityTable of contentsOverviewFeaturesCaveatsUsageBasicsDeep analysisMultilingual analysisUsing as a part of Spacy pipelineCustomizationsConsole ExecutableRESTful web serviceInstallationBasic installationDeep analysisOther language supportRussian language supportPymorphy2Multilingual supportRESTful web serviceTroubleshootingCreditsOverviewprofanity-filteris a universal library for detecting and filtering profanity. Support for English and Russian is
included.FeaturesFull text or individual words censoring.Multilingual support, including profanity filtering in texts written in mixed languages.Deep analysis. The library detects not only the exact profane word matches but also derivative and distorted profane
words using the Levenshtein automata, ignoring dictionary words, containing profane words as a part.Spacy component for using the library as a part of the pipeline.Explanation of decisions (attributeoriginal_profane_word).Partial word censoring.Extensibility support. New languages can be added by supplying dictionaries.RESTful web service.CaveatsContext-free. The library cannot detect using profane phrases consisted of decent words. Vice versa, the library
cannot detect appropriate usage of a profane word.UsageHere are the basic examples of how to use the library. For more examples please seetestsfolder.Basicsfromprofanity_filterimportProfanityFilterpf=ProfanityFilter()pf.censor("That's bullshit!")# "That's ********!"pf.censor_word('fuck')# Word(uncensored='fuck', censored='****', original_profane_word='fuck')Deep analysisfromprofanity_filterimportProfanityFilterpf=ProfanityFilter()pf.censor("fuckfuck")# "********"pf.censor_word('oofuko')# Word(uncensored='oofuko', censored='******', original_profane_word='fuck')pf.censor_whole_words=Falsepf.censor_word('h0r1h0r1')# Word(uncensored='h0r1h0r1', censored='***1***1', original_profane_word='h0r')Multilingual analysisfromprofanity_filterimportProfanityFilterpf=ProfanityFilter(languages=['ru','en'])pf.censor("Да бля, это просто shit какой-то!")# "Да ***, это просто **** какой-то!"Using as a part of Spacy pipelineimportspacyfromprofanity_filterimportProfanityFilternlp=spacy.load('en')profanity_filter=ProfanityFilter(nlps={'en':nlp})# reuse spacy Language (optional)nlp.add_pipe(profanity_filter.spacy_component,last=True)doc=nlp('This is shiiit!')doc._.is_profane# Truedoc[:2]._.is_profane# Falsefortokenindoc:print(f'{token}: 'f'censored={token._.censored}, 'f'is_profane={token._.is_profane}, 'f'original_profane_word={token._.original_profane_word}')# This: censored=This, is_profane=False, original_profane_word=None# is: censored=is, is_profane=False, original_profane_word=None# shiiit: censored=******, is_profane=True, original_profane_word=shit# !: censored=!, is_profane=False, original_profane_word=NoneCustomizationsfromprofanity_filterimportProfanityFilterpf=ProfanityFilter()pf.censor_char='@'pf.censor("That's bullshit!")# "That's @@@@@@@@!"pf.censor_char='*'pf.custom_profane_word_dictionaries={'en':{'love','dog'}}pf.censor("I love dogs and penguins!")# "I **** **** and penguins"pf.restore_profane_word_dictionaries()pf.is_clean("That's awesome!")# Truepf.is_clean("That's bullshit!")# Falsepf.is_profane("That's bullshit!")# Truepf.extra_profane_word_dictionaries={'en':{'chocolate','orange'}}pf.censor("Fuck orange chocolates")# "**** ****** **********"Console Executable$profanity_filter-h
usage:profanity_filter[-h][-tTEXT|-fPATH][-lLANGUAGES][-oOUTPUT_FILE][--show]Profanityfilterconsoleutility

optionalarguments:-h,--helpshowthishelpmessageandexit-tTEXT,--textTEXTTestthegiventextforprofanity-fPATH,--filePATHTestthegivenfileforprofanity-lLANGUAGES,--languagesLANGUAGESTestforprofanityusingspecifiedlanguages(commaseparated)-oOUTPUT_FILE,--outputOUTPUT_FILEWritethecensoredoutputtoafile--showPrintthecensoredtextRESTful web serviceRun:$uvicornprofanity_filter.web:app--reload
INFO:Uvicornrunningonhttp://127.0.0.1:8000
...Go to the{BASE_URL}/docsfor interactive documentation.InstallationFirst two parts of installation instructions are designed for the users who want to filter English profanity.
If you want to filter profanity in another language you still need to read it.Basic installationFor minimal setup you need to installprofanity-filterwith is bundled withspacyand downloadspacymodel for tokenization and lemmatization:$pipinstallprofanity-filter
$# Skip next line if you want to filter profanity in another language$python-mspacydownloadenFor more info about Spacy models read:https://spacy.io/usage/models/.Deep analysisTo get deep analysis functionality install additional libraries and dictionary for your language.Firstly, installhunspellandhunspell-develpackages with your system package manager.For Amazon Linux AMI run:$sudoyuminstallhunspellFor openSUSE run:$sudozypperinstallhunspellhunspell-develThen run:$pipinstall-Uprofanity-filter[deep-analysis]git+https://github.com/rominf/hunspell_serializable@49c00fabf94cacf9e6a23a0cd666aac10cb1d491#egg=hunspell_serializablegit+https://github.com/rominf/pyffs@6c805fbfd7771727138b169b32484b53c0b0fad1#egg=pyffs
$# Skip next lines if you want deep analysis support for another language (will be covered in next section)$cdprofanity_filter/data
$wgethttps://cgit.freedesktop.org/libreoffice/dictionaries/plain/en/en_US.aff
$wgethttps://cgit.freedesktop.org/libreoffice/dictionaries/plain/en/en_US.dic
$mven_US.affen.aff
$mven_US.dicen.dicOther language supportLet's take Russian for example on how to add new language support.Russian language supportFirstly, we need to provide fileprofanity_filter/data/ru_badwords.txtwhich contains a newline separated list of
profane words. For Russian it's already present, so we skip file generation.Next, we need to download the appropriate Spacy model. Unfortunately, Spacy model for Russian is not yet ready, so we
will use an English model for tokenization. If you had not install Spacy model for English, it's the right time to do
so. As a consequence, even if you want to filter just Russian profanity, you need to specify English inProfanityFilterconstructor as shown in usage examples.Next, we download dictionaries in Hunspell format for deep analysis from the sitehttps://cgit.freedesktop.org/libreoffice/dictionaries/plain/:>cdprofanity_filter/data
>wgethttps://cgit.freedesktop.org/libreoffice/dictionaries/plain/ru_RU/ru_RU.aff
>wgethttps://cgit.freedesktop.org/libreoffice/dictionaries/plain/ru_RU/ru_RU.dic
>mvru_RU.affru.aff
>mvru_RU.dicru.dicPymorphy2For Russian and Ukrainian languages to achieve better results we suggest you to installpymorphy2.
To installpymorphy2with Russian dictionary run:$pipinstall-Uprofanity-filter[pymorphy2-ru]git+https://github.com/kmike/pymorphy2@ca1c13f6998ae2d835bdd5033c17197dcba84cf4#egg=pymorphy2Multilingual supportYou need to installpolyglotpackage and it's requirements for language detection.
Seehttps://polyglot.readthedocs.io/en/latest/Installation.htmlfor more detailed instructions.For Amazon Linux AMI run:$sudoyuminstalllibicu-develFor openSUSE run:$sudozypperinstalllibicu-develThen run:$pipinstall-Uprofanity-filter[multilingual]RESTful web serviceRun:$pipinstall-Uprofanity-filter[web]TroubleshootingYou can always check will deep, morphological, and multilingual analyses work by inspecting the value of module
variableAVAILABLE_ANALYSES. If you've followed all steps and installed support for all analyses you will see the
following:fromprofanity_filterimportAVAILABLE_ANALYSESprint(', '.join(sorted(analysis.valueforanalysisinAVAILABLE_ANALYSES)))# deep, morphological, multilingualIf something is not right, you can import dependencies yourself to see the import exceptions:fromprofanity_filter.analysis.deepimport*fromprofanity_filter.analysis.morphologicalimport*fromprofanity_filter.analysis.multilingualimport*CreditsEnglish profane word dictionary:https://github.com/areebbeigh/profanityfilter/(author Areeb Beigh).Russian profane word dictionary:https://github.com/PixxxeL/djantimat(author Ivan Sergeev).