Package: html-text
HTML to TextExtract text from HTMLFree software: MIT licenseHow is html_text different from.xpath('//text()')from LXML
or.get_text()from Beautiful Soup?Text extracted withhtml_textdoes not contain inline styles,
javascript, comments and other text that is not normally visible to users;html_textnormalizes whitespace, but in a way smarter than.xpath('normalize-space()), adding spaces around inline elements
(which are often used as block elements in html markup), and trying to
avoid adding extra spaces for punctuation;html-textcan add newlines (e.g. after headers or paragraphs), so
that the output text looks more like how it is rendered in browsers.InstallInstall with pip:pip install html-textThe package depends on lxml, so you might need to install additional
packages:http://lxml.de/installation.htmlUsageExtract text from HTML:>>> import html_text
>>> html_text.extract_text('<h1>Hello</h1> world!')
'Hello\n\nworld!'

>>> html_text.extract_text('<h1>Hello</h1> world!', guess_layout=False)
'Hello world!'Passed html is first cleaned from invisible non-text content such
as styles, and then text is extracted.You can also pass an already parsedlxml.html.HtmlElement:>>> import html_text
>>> tree = html_text.parse_html('<h1>Hello</h1> world!')
>>> html_text.extract_text(tree)
'Hello\n\nworld!'If you want, you can handle cleaning manually; use lower-levelhtml_text.etree_to_textin this case:>>> import html_text
>>> tree = html_text.parse_html('<h1>Hello<style>.foo{}</style>!</h1>')
>>> cleaned_tree = html_text.cleaner.clean_html(tree)
>>> html_text.etree_to_text(cleaned_tree)
'Hello!'parsel.Selector objects are also supported; you can define
a parsel.Selector to extract text only from specific elements:>>> import html_text
>>> sel = html_text.cleaned_selector('<h1>Hello</h1> world!')
>>> subsel = sel.xpath('//h1')
>>> html_text.selector_to_text(subsel)
'Hello'NB parsel.Selector objects are not cleaned automatically, you need to callhtml_text.cleaned_selectorfirst.Main functions and objects:html_text.extract_textaccepts html and returns extracted text.html_text.etree_to_textaccepts parsed lxml Element and returns
extracted text; it is a lower-level function, cleaning is not handled
here.html_text.cleaneris anlxml.html.clean.Cleanerinstance which
can be used withhtml_text.etree_to_text; its options are tuned for
speed and text extraction quality.html_text.cleaned_selectoraccepts html as text or aslxml.html.HtmlElement, and returns cleanedparsel.Selector.html_text.selector_to_textacceptsparsel.Selectorand returns
extracted text.Ifguess_layoutis True (default), a newline is added before and afternewline_tags, and two newlines are added before and afterdouble_newline_tags. This heuristic makes the extracted text
more similar to how it is rendered in the browser. Default newline and double
newline tags can be found inhtml_text.NEWLINE_TAGSandhtml_text.DOUBLE_NEWLINE_TAGS.It is possible to customize how newlines are added, usingnewline_tagsanddouble_newline_tagsarguments (which arehtml_text.NEWLINE_TAGSandhtml_text.DOUBLE_NEWLINE_TAGSby default). For example, don’t add a newline
after<div>tags:>>> newline_tags = html_text.NEWLINE_TAGS - {'div'}
>>> html_text.extract_text('<div>Hello</div> world!',
...                        newline_tags=newline_tags)
'Hello world!'Apart from just getting text from the page (e.g. for display or search),
one intended usage of this library is for machine learning (feature extraction).
If you want to use the text of the html page as a feature (e.g. for classification),
this library gives you plain text that you can later feed into a standard text
classification pipeline.
If you feel that you need html structure as well, check outwebstructlibrary.History0.5.2 (2020-07-22)Handle lxml Cleaner exceptions (a workaround forhttps://bugs.launchpad.net/lxml/+bug/1838497);Python 3.8 support;testing improvements.0.5.1 (2019-05-27)Fixed whitespace handling whenguess_punct_spaceis False: html-text was
producing unnecessary spaces after newlines.0.5.0 (2018-11-19)Parsel dependency is removed in this release,
though parsel is still supported.parselpackage is no longer required to install and use html-text;html_text.etree_to_textfunction allows to extract text from
lxml Elements;html_text.cleaneris anlxml.html.clean.Cleanerinstance with
options tuned for text extraction speed and quality;test and documentation improvements;Python 3.7 support.0.4.1 (2018-09-25)Fixed a regression in 0.4.0 release: text was empty whenhtml_text.extract_textis called with a node with text, but
without children.0.4.0 (2018-09-25)This is a backwards-incompatible release: by default html_text functions
now add newlines after elements, if appropriate, to make the extracted text
to look more like how it is rendered in a browser.To turn it off, passguess_layout=Falseoption to html_text functions.guess_layoutoption to to make extracted text look more like how
it is rendered in browser.Add tests of layout extraction for real webpages.0.3.0 (2017-10-12)Expose functions that operate on selectors,
use.//text()to extract text from selector.0.2.1 (2017-05-29)Packaging fix (include CHANGES.rst)0.2.0 (2017-05-29)Fix unwanted joins of words with inline tags: spaces are added for inline
tags too, but a heuristic is used to preserve punctuation without extra spaces.Accept parsed html trees.0.1.1 (2017-01-16)Travis-CI and codecov.io integrations added0.1.0 (2016-09-27)First release on PyPI.