Package: readabilipy
ReadabiliPyReadabiliPycontains a Python wrapper for Mozilla'sReadability.jsNode.js package, as well as article extraction routines written in pure Python.This package augments the output ofReadability.jsto also return a list of plain text representations of article paragraphs.ReadabiliPycomes with a handy command line application:readabilipy.InstallationTo use theReadability.jswrapper you need to have a workingNode.jsinstallation of version 10 or higher.
Make sure to install Node.js before installing this package, as this ensures Readability.js will be installed.
If you only want to use the Python-based article extraction, youdo not needto install Node.js.ReadabiliPycan be installed simply from PyPI:$ pip install readabilipyNote that to update to a new version ofReadability.jsyou can simply reinstallReadabiliPy.UsageReadabiliPycan be used either as a command line application or as a Python library.Command line applicationThereadabilipycommand line application can be used to extract an article from an HTML source file.For example, if you have the article saved asinput.htmlin the current directory then you can run:$ readabilipy -i ./input.html -o article.jsonThe extracted article can then be found in thearticle.jsonfile. By default ReadabiliPy will use the Readability.js functionality to extract the article, provided this is available. If instead you'd like to use the Python-based extraction, run:$ readabilipy -p -i ./input.html -o article.jsonThe complete help text of the command line application is as follows:$ readabilipy -h
usage: readabilipy [-h] -i INPUT_FILE -o OUTPUT_FILE [-c] [-n] [-p] [-V]

Extract article data from a HTML file using either Mozilla's Readability.js
package or a simplified python-only alternative.

optional arguments:
  -h, --help            show this help message and exit
  -i INPUT_FILE, --input-file INPUT_FILE
                        Path to input file containing HTML.
  -o OUTPUT_FILE, --output-file OUTPUT_FILE
                        Path to file to output the article data to as JSON.
  -c, --content-digests
                        Add a 'data-content-digest' attribute containing a
                        SHA256-based digest of the element's contents to each
                        HTML element in the plain_content output.
  -n, --node-indexes    Add a 'data-node-index' attribute containing a
                        hierarchical representation of the element's position
                        in the HTML structure each HTML element in the
                        plain_content output.
  -p, --use-python-parser
                        Use the pure-python 'plain_html' parser included in
                        this project rather than Mozilla's Readability.js.
  -V, --version         Show version and exitLibraryReadabiliPy can also be used as a Python package.
The main routine is calledsimple_json_from_html_stringand expects the HTML article as a string.
Here is an example of extracting an article after downloading the page usingrequests:>>>importrequests>>>fromreadabilipyimportsimple_json_from_html_string>>>req=requests.get('https://en.wikipedia.org/wiki/Readability')>>>article=simple_json_from_html_string(req.text,use_readability=True)Note that you need to use the flaguse_readability=Trueto use Readability.js, otherwise the Python-based extraction is used.Thesimple_json_from_html_stringfunction returns a dictionary with the following fields:title: The article titlebyline: Author informationcontent: A simplified HTML representation of the article, with all article text contained in paragraph elements.plain_content: A "plain" version of the simplifiedReadability.jsarticle HTML present in thecontentfield. This attempts to retain only the plain text content of the article, while preserving the HTML structure.plain_text: A list containing plain text representations of each paragraph (<p>) or list (<ol>or<ul>) present in the simplifiedReadability.jsarticle HTML in thecontentfield. Each paragraph or list is represented as a single string. List strings look like"* item 1, * item 2, * item 3,"for both ordered and unordered lists (note the trailing,).Note further that:All fields are guaranteed to be present. If individual fields are missing from the output ofReadability.js, the value of these fields will beNone. If no article data is returned byReadability.js, the value of all fields will beNone.All text in theplain_contentandplain_textfields is encoded as unicode normalised using the "NFKC" normal form. This normal form is used to try and ensure as much as possible that things that appear visually the same are encoded with the same unicode representation (the K part) and characters are represented as a single composite character where possible (the C part).An optionalcontent_digestsflag can be passed to the Python wrapper. When this is set toTrue, each HTML element in theplain_contentfield has adata-content-digestattribute, which holds the SHA-256 hash of its plain text content. For "leaf" nodes (containing only plain text in the output), this is the SHA-256 hash of their plain text content. For nodes containing other nodes, this is the SHA-256 hash of the concatenated SHA-256 hashes of their child nodes.An optionalnode_indexesflag can be passed to the Python wrapper. When this is set toTrue, each HTML element in theplain_contentfield has adata-node-indexesattribute, which holds a hierarchical index describing the location of element within theplain_contentHTML structure.An optionaluse_readabilityflag can be passed to the Python wrapper. When this is set toTrue, Mozilla'sReadability.jswill be used as the parser. If it is set toFalsethen the pure-python parser inplain_html.pywill be used instead.The second top-level function exported by ReadabiliPy issimple_tree_from_html_string. This returns a cleaned, parsed HTML tree of the article as aBeautifulSoupobject.NotesLicense: MIT License, see theLICENSEfile.Copyright (c) 2018, The Alan Turing InstituteIf you encounter any issues or have any suggestions for improvement, please open an issueon Github.
You're helping to make this project better for everyone!