Package: html5lib
html5lib is a pure-python library for parsing HTML. It is designed to
conform to the WHATWG HTML specification, as is implemented by all major
web browsers.UsageSimple usage follows this pattern:importhtml5libwithopen("mydocument.html","rb")asf:document=html5lib.parse(f)or:importhtml5libdocument=html5lib.parse("<p>Hello World!")By default, thedocumentwill be anxml.etreeelement instance.
Whenever possible, html5lib chooses the acceleratedElementTreeimplementation (i.e.xml.etree.cElementTreeon Python 2.x).Two other tree types are supported:xml.dom.minidomandlxml.etree. To use an alternative format, specify the name of
a treebuilder:importhtml5libwithopen("mydocument.html","rb")asf:lxml_etree_document=html5lib.parse(f,treebuilder="lxml")When using withurllib2(Python 2), the charset from HTTP should be
pass into html5lib as follows:fromcontextlibimportclosingfromurllib2importurlopenimporthtml5libwithclosing(urlopen("http://example.com/"))asf:document=html5lib.parse(f,transport_encoding=f.info().getparam("charset"))When using withurllib.request(Python 3), the charset from HTTP
should be pass into html5lib as follows:fromurllib.requestimporturlopenimporthtml5libwithurlopen("http://example.com/")asf:document=html5lib.parse(f,transport_encoding=f.info().get_content_charset())To have more control over the parser, create a parser object explicitly.
For instance, to make the parser raise exceptions on parse errors, use:importhtml5libwithopen("mydocument.html","rb")asf:parser=html5lib.HTMLParser(strict=True)document=parser.parse(f)When you’re instantiating parser objects explicitly, pass a treebuilder
class as thetreekeyword argument to use an alternative document
format:importhtml5libparser=html5lib.HTMLParser(tree=html5lib.getTreeBuilder("dom"))minidom_document=parser.parse("<p>Hello World!")More documentation is available athttps://html5lib.readthedocs.io/.Installationhtml5lib works on CPython 2.7+, CPython 3.5+ and PyPy. To install:$pipinstallhtml5libThe goal is to support a (non-strict) superset of the versions thatpip
supports.Optional DependenciesThe following third-party libraries may be used for additional
functionality:lxmlis supported as a tree format (for both building and
walking) under CPython (butnotPyPy where it is known to cause
segfaults);genshihas a treewalker (but not builder); andchardetcan be used as a fallback when character encoding cannot
be determined.BugsPlease report any bugs on theissue tracker.TestsUnit tests require thepytestandmocklibraries and can be
run using thepy.testcommand in the root directory.Test data are contained in a separatehtml5lib-testsrepository and included
as a submodule, thus for git checkouts they must be initialized:$ git submodule init
$ git submodule updateIf you have all compatible Python implementations available on your
system, you can run tests on all of them using thetoxutility,
which can be found on PyPI.Questions?There’s a mailing list available for support on Google Groups,html5lib-discuss,
though you may get a quicker response asking on IRC in#whatwg on
irc.freenode.net.Change Log1.1UNRELEASEDBreaking changes:Drop support for Python 3.3. (#358)Drop support for Python 3.4. (#421)Deprecations:Deprecate thehtml5libsanitizer (html5lib.serialize(sanitize=True)andhtml5lib.filters.sanitizer). We recommend users migrate toBleach
<https://github.com/mozilla/bleach>. Please let us know if Bleach doesn’t suffice for your
use. (#443)Other changes:Try to import fromcollections.abcto remove DeprecationWarning and ensurehtml5libkeeps working in future Python versions. (#403)Drop optionaldatriedependency. (#442)1.0.1Released on December 7, 2017Breaking changes:Drop support for Python 2.6. (#330) (Thank you, Hugo, Will Kahn-Greene!)Removeutils/spider.py(#353) (Thank you, Jon Dufresne!)Features:Improve documentation. (#300, #307) (Thank you, Jon Dufresne, Tom Most,
Will Kahn-Greene!)Add iframe seamless boolean attribute. (Thank you, Ritwik Gupta!)Add itemscope as a boolean attribute. (#194) (Thank you, Jonathan Vanasco!)Support Python 3.6. (#333) (Thank you, Jon Dufresne!)Add CI support for Windows using AppVeyor. (Thank you, John Vandenberg!)Improve testing and CI and add code coverage (#323, #334), (Thank you, Jon
Dufresne, John Vandenberg, Sam Sneddon, Will Kahn-Greene!)Semver-compliant version number.Bug fixes:Add support for setuptools < 18.5 to support environment markers. (Thank you,
John Vandenberg!)Add explicit dependency for six >= 1.9. (Thank you, Eric Amorde!)Fix regexes to work with Python 3.7 regex adjustments. (#318, #379) (Thank
you, Benedikt Morbach, Ville Skyttä, Mark Vasilkov!)Fix alphabeticalattributes filter namespace bug. (#324) (Thank you, Will
Kahn-Greene!)Include license file in generated wheel package. (#350) (Thank you, Jon
Dufresne!)Fix annotation-xml typo. (#339) (Thank you, Will Kahn-Greene!)Allow uppercase hex chararcters in CSS colour check. (#377) (Thank you,
Komal Dembla, Hugo!)1.0Released and unreleased on December 7, 2017. Badly packaged release.0.999999999/1.0b10Released on July 15, 2016Fix attribute order going to the tree builder to be document order
instead of reverse document order(!).0.99999999/1.0b9Released on July 14, 2016Added ordereddict as a mandatory dependency on Python 2.6.Addedlxml,genshi,datrie,charade, andallextras that will do the right thing based on the specific
interpreter implementation.Now requires themockpackage for the testsuite.Cease supporting DATrie under PyPy.Remove PullDOM support, as this hasn’t ever been properly
tested, doesn’t entirely work, and as far as I can tell is
completely unused by anyone.Move testsuite topy.test.Fix #124: move to webencodings for decoding the input byte stream;
this makes html5lib compliant with the Encoding Standard, and
introduces a required dependency on webencodings.Cease supporting Python 3.2 (in both CPython and PyPy forms).Fix comments containing double-dash with lxml 3.5 and above.Use scripting disabled by default (as we don’t implement
scripting).Fix #11, avoiding the XSS bug potentially caused by serializer
allowing attribute values to be escaped out of in old browser versions,
changing the quote_attr_values option on serializer to take one of
three values, “always” (the old True value), “legacy” (the new option,
and the new default), and “spec” (the old False value, and the old
default).Fix #72 by rewriting the sanitizer to apply only to treewalkers
(instead of the tokenizer); as such, this will require amending all
callers of it to use it via the treewalker API.Drop support of charade, now that chardet is supported once more.Replace the charset keyword argument on parse and related methods
with a set of keyword arguments: override_encoding, transport_encoding,
same_origin_parent_encoding, likely_encoding, and default_encoding.Move filters._base, treebuilder._base, and treewalkers._base to .base
to clarify their status as public.Get rid of the sanitizer package. Merge sanitizer.sanitize into the
sanitizer.htmlsanitizer module and move that to sanitizer. This means
anyone who used sanitizer.sanitize or sanitizer.HTMLSanitizer needs no
code changes.Rename treewalkers.lxmletree to .etree_lxml and
treewalkers.genshistream to .genshi to have a consistent API.Move a whole load of stuff (inputstream, ihatexml, trie, tokenizer,
utils) to be underscore prefixed to clarify their status as private.0.9999999/1.0b8Released on September 10, 2015Fix #195: fix the sanitizer to drop broken URLs (it threw an
exception between 0.9999 and 0.999999).0.999999/1.0b7Released on July 7, 2015Fix #189: fix the sanitizer to allow relative URLs again (as it did
prior to 0.9999/1.0b5).0.99999/1.0b6Released on April 30, 2015Fix #188: fix the sanitizer to not throw an exception when sanitizing
bogus data URLs.0.9999/1.0b5Released on April 29, 2015Fix #153: Sanitizer fails to treat some attributes as URLs. Despite how
this sounds, this has no known security implications.  No known version
of IE (5.5 to current), Firefox (3 to current), Safari (6 to current),
Chrome (1 to current), or Opera (12 to current) will run any script
provided in these attributes.Pass error message to the ParseError exception in strict parsing mode.Allow data URIs in the sanitizer, with a whitelist of content-types.Add support for Python implementations that don’t support lone
surrogates (read: Jython). Fixes #2.Remove localization of error messages. This functionality was totally
unused (and untested that everything was localizable), so we may as
well follow numerous browsers in not supporting translating technical
strings.Expose treewalkers.pprint as a public API.Add a documentEncoding property to HTML5Parser, fix #121.0.999Released on December 23, 2013Fix #127: add work-around for CPython issue #20007: .read(0) on
http.client.HTTPResponse drops the rest of the content.Fix #115: lxml treewalker can now deal with fragments containing, at
their root level, text nodes with non-ASCII characters on Python 2.0.99Released on September 10, 2013No library changes from 1.0b3; released as 0.99 as pip has changed
behaviour from 1.4 to avoid installing pre-release versions per
PEP 440.1.0b3Released on July 24, 2013RemovedRecursiveTreeWalkerfromtreewalkers._base. Any
implementation using it should be moved toNonRecursiveTreeWalker, as everything bundled with html5lib has
for years.Fix #67 so thatBufferedStreamto correctly returns a bytes
object, thereby fixing any case where html5lib is passed a
non-seekable RawIOBase-like object.1.0b2Released on June 27, 2013Removed reordering of attributes within the serializer. There is now
analphabetical_attributesoption which preserves the previous
behaviour through a new filter. This allows attribute order to be
preserved through html5lib if the tree builder preserves order.Removeddom2saxfrom DOM treebuilders. It has been replaced bytreeadapters.sax.to_saxwhich is generic and supports any
treewalker; it also resolves all known bugs withdom2sax.Fix treewalker assertions on hitting bytes strings on
Python 2. Previous to 1.0b1, treewalkers coped with mixed
bytes/unicode data on Python 2; this reintroduces this prior
behaviour on Python 2. Behaviour is unchanged on Python 3.1.0b1Released on May 17, 2013Implementation updated to implement theHTML specificationas of 5th May
2013 (SVNrevision r7867).Python 3.2+ supported in a single codebase using thesixlibrary.Removed support for Python 2.5 and older.Removed the deprecated Beautiful Soup 3 treebuilder.beautifulsoup4can usehtml5libas a parser instead. Note that
since it doesn’t support namespaces, foreign content like SVG and
MathML is parsed incorrectly.Removedsimpletreefrom the package. The default tree builder is
nowetree(using thexml.etree.cElementTreeimplementation if
available, andxml.etree.ElementTreeotherwise).Removed theXHTMLSerializeras it never actually guaranteed its
output was well-formed XML, and hence provided little of use.Removed default DOM treebuilder, sohtml5lib.treebuilders.domis no
longer supported.html5lib.treebuilders.getTreeBuilder("dom")will
return the default DOM treebuilder, which usesxml.dom.minidom.Optional heuristic character encoding detection now based oncharadefor Python 2.6 - 3.3 compatibility.OptionalGenshitreewalker support fixed.Many bugfixes, including:#33: null in attribute value breaks XML AttValue;#4: nested, indirect descendant, <button> causes infinite loop;Google Code 215: Properly
detect seekable streams;Google Code 206: add
support for <video preload=…>, <audio preload=…>;Google Code 205: add
support for <video poster=…>;Google Code 202: Unicode
file breaks InputStream.Source code is now mostly PEP 8 compliant.Test harness has been improved and now depends onnose.Documentation updated and moved tohttps://html5lib.readthedocs.io/.0.95Released on February 11, 20120.90Released on January 17, 20100.11.1Released on June 12, 20080.11Released on June 10, 20080.10Released on October 7, 20070.9Released on March 11, 20070.2Released on January 8, 2007