Package: junkdetect
Junk, Not-Junk DetectorThis tool is built to do just one simple task: detect junk and not-junk texts from a variety of languages.
Just like that famoushotdog not-hotdog, but applied on natural language text.
It can be very useful to test tools that extract, decompress, and/or decrypt natural language texts.SetupUsesfairseq# Optionally create a brand new conda environment for this#conda create -n junkdetect python=3.7#conda activate junkdetect# Install: use only one of these methods# 1. from pypi; recommendedpipinstalljunkdetect# 2. latest master branchpipinstallgit+https://github.com/thammegowda/junkdetect# 3. for developmentgitclonehttps://github.com/thammegowda/junkdetect\&&cdjunkdetect\&&pipinstall--editable.How to useOnce you install it via pip,junkdetectorpython -m junkdetectcan be used to invoke from commandlineprintf"This is a good sentence. \nT6785*&^T is 747658 you T&*^\n"|junkdetect0.999824Thisisagoodsentence.0.0747487T6785*&^Tis747658youT&*^The output is one line per input, with two column separated  by\t.
The first column hasperplexity: a lower value (i.e close to 0.0) means junk and an higher value (close to 1.0) means not-junk. If you dont want input sentences back in the output, please cut them out -- just usejunkdetect | cut -f1 > scores.txtHow does this workjunkdetectlooks like only a few lines of python code, but under the hood, it hides a great deal of complexity.It uses perplexity from neural (masked/auto-regressive) language models that were trained on tera bytes of web text from 100s of languages.Specifically, it uses Facebookresearch'sXML-Rretrieved fromtorch.hub.
Quoting the original developers of XML-R andtheir paper, (see Table 6)XLM-R handles the following 100 languages: Afrikaans, Albanian, Amharic, Arabic, Armenian, Assamese, Azerbaijani, Basque, Belarusian, Bengali, Bengali Romanized, Bosnian, Breton, Bulgarian, Burmese, Burmese, Catalan, Chinese (Simplified), Chinese (Traditional), Croatian, Czech, Danish, Dutch, English, Esperanto, Estonian, Filipino, Finnish, French, Galician, Georgian, German, Greek, Gujarati, Hausa, Hebrew, Hindi, Hindi Romanized, Hungarian, Icelandic, Indonesian, Irish, Italian, Japanese, Javanese, Kannada, Kazakh, Khmer, Korean, Kurdish (Kurmanji), Kyrgyz, Lao, Latin, Latvian, Lithuanian, Macedonian, Malagasy, Malay, Malayalam, Marathi, Mongolian, Nepali, Norwegian, Oriya, Oromo, Pashto, Persian, Polish, Portuguese, Punjabi, Romanian, Russian, Sanskri, Scottish, Gaelic, Serbian, Sindhi, Sinhala, Slovak, Slovenian, Somali, Spanish, Sundanese, Swahili, Swedish, Tamil, Tamil Romanized, Telugu, Telugu Romanized, Thai, Turkish, Ukrainian, Urdu, Urdu Romanized, Uyghur, Uzbek, Vietnamese, Welsh, Western, Frisian, Xhosa, Yiddish.Back Story and Acknowledgements:This idea came out of discussion withTim Allison.
He said it was hard to tell whether text was correctly extracted or not from files like PDFs using Apache Tika.
Thanks to him for making me think of something like this.I had read Facebook's very niceXML-R paper of Conneau et aland it was top of my mind.
Although XLM folksdidnt help me get perplexity, and I had to dug it out of their code by myself,
I still like to thank them for making such useful pretrained models available for easy to use viatorch.hub.Developers:Thamme Gowda(wrote the version 0.1)