Package: jusText-lid
@jordiae ChangesUse Fasttext language identifier instead of stopwords.Install:pipinstalljustext-lidUsage (no stopwords required):fromjustext_lidimportjustextfirst_paragraph=justext(HTML,langs={'en','es'},langid_th=0.3)[0]first_paragraph_lang=first_paragraph.langEleutherAI ChangesModified the classification function in order to remove elements that have quotes (in order to improve results on forums)jusTextProgram jusText is a tool for removing boilerplate content, such as navigation
links, headers, and footers from HTML pages. It isdesignedto preserve
mainly text containing full sentences and it is therefore well suited for
creating linguistic resources such as Web corpora. You cantry it online.This is a fork of original (currently unmaintained) code ofjusTexthosted
on Google Code.Adaptations of the algorithm to other languages:C++GoJavaSome libraries using jusText:chirplazynlpoff-topic-memento-toolkitpearsreadability calculatorskySome currently (Jan 2020) maintained alternatives:dragnethtml2textinscriptisnewspaperpython-readabilitytrafilaturaInstallationMake sure you havePython2.7+/3.4+ andpip(Windows,Linux) installed.
Run simply:$[sudo]pipinstalljustextDependencieslxml (version depends on your Python version)Usage$python-mjustext-sCzech-otext.txthttp://www.zdrojak.cz/clanky/automaticke-zabezpeceni/$python-mjustext-sEnglish-oplain_text.txtenglish_page.html$python-mjustext--help# for more infoPython APIimportrequestsimportjustextresponse=requests.get("http://planet.python.org/")paragraphs=justext.justext(response.content,justext.get_stoplist("English"))forparagraphinparagraphs:ifnotparagraph.is_boilerplate:printparagraph.textTestingRun tests via$py.test-2.7&&py.test-3.4&&py.test-3.5&&py.test-3.6&&py.test-3.7&&py.test-3.8AcknowledgementsThis software has been developed at theNatural Language Processing CentreofMasaryk University in Brnowith a financial support fromPRESEMTandLexical Computing Ltd.It also relates toPhD researchof Jan Pomikálek.Changelog for jusText2.2.0 (2016-03-06)INCOMPATIBLE CHANGE:Stop words are case insensitive.INCOMPATIBLE CHANGE:Dropped support for Python 3.2BUG FIX:Preserve new lines from original text in paragraphs.2.1.1 (2014-05-27)BUG FIX:Functiondecode_htmlnow respects parametererrorswhen falling todefault_encoding#9.2.1.0 (2014-01-25)FEATURE:Added XPath selector to the paragrahs. XPath selector is also available in detailed output asxpathattribute of<p>tag#5.2.0.0 (2013-08-26)FEATURE:Added pluggable DOM preprocessor.FEATURE:Added support for Python 3.2+.INCOMPATIBLE CHANGE:Paragraphs are instances ofjustext.paragraph.Paragraph.INCOMPATIBLE CHANGE:Script ‘justext’ removed in favour of
commandpython-mjustext.FEATURE:It’s possible to enter an URI as input document in CLI.FEATURE:It is possible to pass unicode string directly.1.2.0 (2011-08-08)FEATURE:Character counts used instead of word counts where possible in
order to make the algorithm work well in the language independent
mode (without a stoplist) for languages where counting words is
not easy (Japanese, Chinese, Thai, etc).BUG FIX:More robust parsing of meta tags containing the information about
used charset.BUG FIX:Corrected decoding of HTML entities &#128; to &#159;1.1.0 (2011-03-09)First public release.