Package: inscriptis
A python based HTML to text conversion library, command line client and Web
service with support fornested tables, asubset of CSSand optional
support for providing anannotated output.Inscriptis is particularly well suited for applications that require high-performance, high-quality (i.e., layout-aware) text representations of HTML content, and will aid knowledge extraction and data science tasks conducted upon Web data.Please take a look at theRenderingdocument for a demonstration of inscriptis’ conversion quality.A Java port of inscriptis 1.x has been published byx28.This document provides a short introduction to Inscriptis.The full documentation is built automatically and published onRead the Docs.If you are interested in a more general overview on the topic oftext extraction from HTML, thisblog post on different HTML to text conversion approaches, and criteria for selecting themmight be interesting to you.Table of contentsStatement of need - why inscriptis?InstallationPython libraryStandalone command line clientCommand line parametersHTML to text conversionHTML to annotated text conversionAnnotation postprocessorsWeb ServiceRun the Web Service on your host systemRun the Web Service with DockerRun as Kubernetes DeploymentUse the Web ServiceExample annotation profilesWikipedia tables and table metadataReferences to entities, missing entities and citations from WikipediaPosts and post metadata from the XDA developer forumCode and metadata from Stackoverflow pagesAdvanced topicsAnnotated textFine tuningCustom HTML tag handlingOptimizing memory consumptionExamplesStrict indentation handlingIgnore elements during parsingCitationChangelogStatement of need - why inscriptis?Inscriptis provides alayout-awareconversion of HTML that more closely resembles the rendering obtained from standard Web browsers and, therefore, better preserves the spatial arrangement of text elements.Conversion quality becomes a factor once you need to move beyond simple HTML snippets. Non-specialized approaches and less sophisticated libraries do not correctly interpret HTML semantics and, therefore, fail to properly convert constructs such as itemizations, enumerations, and tables.Beautiful Soup’sget_text()function, for example, converts the following HTML enumeration to the stringfirstsecond.<ul><li>first</li><li>second</li><ul>Inscriptis, in contrast, not only returns the correct output* first
* secondbut also supports much more complex constructs such as nested tables and also interprets a subset of HTML (e.g.,align,valign) and CSS (e.g.,display,white-space,margin-top,vertical-align, etc.) attributes that determine the text alignment. Any time the spatial alignment of text is relevant (e.g., for many knowledge extraction tasks, the computation of word embeddings and language models, and sentiment analysis) an accurate HTML to text conversion is essential.Inscriptis supportsannotation rules, i.e., user-provided mappings that allow for annotating the extracted text based on structural and semantic information encoded in HTML tags and attributes used for controlling structure and layout in the original HTML document. These rules might be used toprovide downstream knowledge extraction components with additional information that may be leveraged to improve their respective performance.assist manual document annotation processes (e.g., for qualitative analysis or gold standard creation).Inscriptissupports multiple export formats such as XML, annotated HTML and the JSONL format that is used by the open source annotation tooldoccano.enabling the use ofInscriptisfor tasks such as content extraction (i.e., extract task-specific relevant content from a Web page) which rely on information on the HTML document’s structure.InstallationAt the command line:$ pip install inscriptisOr, if you don’t have pip installed:$ easy_install inscriptisIf you want to install from the latest sources, you can do:$ git clone https://github.com/weblyzard/inscriptis.git
$ cd inscriptis
$ python setup.py installPython libraryEmbedding inscriptis into your code is easy, as outlined below:importurllib.requestfrominscriptisimportget_texturl="https://www.fhgr.ch"html=urllib.request.urlopen(url).read().decode('utf-8')text=get_text(html)print(text)Standalone command line clientThe command line client converts HTML files or text retrieved from Web pages to
the corresponding text representation.Command line parametersThe inscript command line client supports the following parameters:usage: inscript [-h] [-o OUTPUT] [-e ENCODING] [-i] [-d] [-l] [-a] [-r ANNOTATION_RULES] [-p POSTPROCESSOR] [--indentation INDENTATION]
                   [--table-cell-separator TABLE_CELL_SEPARATOR] [-v]
                   [input]

Convert the given HTML document to text.

positional arguments:
  input                 Html input either from a file or a URL (default:stdin).

optional arguments:
  -h, --help            show this help message and exit
  -o OUTPUT, --output OUTPUT
                        Output file (default:stdout).
  -e ENCODING, --encoding ENCODING
                        Input encoding to use (default:utf-8 for files; detected server encoding for Web URLs).
  -i, --display-image-captions
                        Display image captions (default:false).
  -d, --deduplicate-image-captions
                        Deduplicate image captions (default:false).
  -l, --display-link-targets
                        Display link targets (default:false).
  -a, --display-anchor-urls
                        Display anchor URLs (default:false).
  -r ANNOTATION_RULES, --annotation-rules ANNOTATION_RULES
                        Path to an optional JSON file containing rules for annotating the retrieved text.
  -p POSTPROCESSOR, --postprocessor POSTPROCESSOR
                        Optional component for postprocessing the result (html, surface, xml).
  --indentation INDENTATION
                        How to handle indentation (extended or strict; default: extended).
  --table-cell-separator TABLE_CELL_SEPARATOR
                        Separator to use between table cells (default: three spaces).
  -v, --version         display version informationHTML to text conversionconvert the given page to text and output the result to the screen:$ inscript https://www.fhgr.chconvert the file to text and save the output to fhgr.txt:$ inscript fhgr.html -o fhgr.txtconvert the file using strict indentation (i.e., minimize indentation and extra spaces) and save the output to fhgr-layout-optimized.txt:$ inscript --indentation strict fhgr.html -o fhgr-layout-optimized.txtconvert HTML provided via stdin and save the output to output.txt:$ echo "<body><p>Make it so!</p></body>" | inscript -o output.txtHTML to annotated text conversionconvert and annotate HTML from a Web page using the provided annotation rules.Download the exampleannotation-profile.jsonand save it to your working directory:$ inscript https://www.fhgr.ch -r annotation-profile.jsonThe annotation rules are specified inannotation-profile.json:{"h1":["heading","h1"],"h2":["heading","h2"],"b":["emphasis"],"div#class=toc":["table-of-contents"],"#class=FactBox":["fact-box"],"#cite":["citation"]}The dictionary maps an HTML tag and/or attribute to the annotations
inscriptis should provide for them. In the example above, for instance, the tagh1yields the annotationsheadingandh1, adivtag with aclassthat contains the valuetocresults in the annotationtable-of-contents, and all tags with aciteattribute are annotated withcitation.Given these annotation rules the HTML file<h1>Chur</h1><b>Chur</b>is the capital and largest town of the Swiss canton of the
Grisons and lies in the Grisonian Rhine Valley.yields the following JSONL output{"text":"Chur\n\nChur is the capital and largest town of the Swiss canton
          of the Grisons and lies in the Grisonian Rhine Valley.","label":[[0,4,"heading"],[0,4,"h1"],[6,10,"emphasis"]]}The provided list of labels contains all annotated text elements with their
start index, end index and the assigned label.Annotation postprocessorsAnnotation postprocessors enable the post processing of annotations to formats
that are suitable for your particular application. Post processors can be
specified with the-por--postprocessorcommand line argument:$ inscript https://www.fhgr.ch \
        -r ./examples/annotation-profile.json \
        -p surfaceOutput:{"text":"  Chur\n\n  Chur is the capital and largest town of the Swiss
          canton of the Grisons and lies in the Grisonian Rhine Valley.","label":[[0,6,"heading"],[8,14,"emphasis"]],"tag":"<heading>Chur</heading>\n\n<emphasis>Chur</emphasis> is the
        capital and largest town of the Swiss canton of the Grisons and
        lies in the Grisonian Rhine Valley."}Currently, inscriptis supports the following postprocessors:surface: returns a list of mapping between the annotation’s surface form and its label:[
   ['heading', 'Chur'],
   ['emphasis': 'Chur']
]xml: returns an additional annotated text version:<?xml version="1.0" encoding="UTF-8" ?>
<heading>Chur</heading>

<emphasis>Chur</emphasis> is the capital and largest town of the Swiss
canton of the Grisons and lies in the Grisonian Rhine Valley.html: creates an HTML file which contains the converted text and highlights all annotations as outlined below:Snippet of the rendered HTML file created with the following command line options and annotation rules:inscript--annotation-rules./wikipedia.json\--postprocessorhtml\https://en.wikipedia.org/wiki/Chur.htmlAnnotation rules encoded in thewikipedia.jsonfile:{"h1":["heading"],"h2":["heading"],"h3":["subheading"],"h4":["subheading"],"h5":["subheading"],"i":["emphasis"],"b":["bold"],"table":["table"],"th":["tableheading"],"a":["link"]}Web ServiceA FastAPI-based Web Service that uses Inscriptis for translating HTML pages to plain text.Run the Web Service on your host systemInstall the optional featureweb-servicefor inscriptis:$ pip install inscriptis[web-service]Start the Inscriptis Web service with the following command:$ uvicorn inscriptis.service.web:app --port 5000 --host 127.0.0.1Run the Web Service with DockerThe docker definition can be foundhere:$ docker pull ghcr.io/weblyzard/inscriptis:latest
$ docker run -n inscriptis ghcr.io/weblyzard/inscriptis:latestRun as Kubernetes DeploymentThe helm chart for deployment on a kubernetes cluster is located in theinscriptis-helm repository.Use the Web ServiceThe Web services receives the HTML file in the request body and returns the
corresponding text. The file’s encoding needs to be specified
in theContent-Typeheader (UTF-8in the example below):$ curl -X POST  -H "Content-Type: text/html; encoding=UTF8"  \
        --data-binary @test.html  http://localhost:5000/get_textThe service also supports a version call:$ curl http://localhost:5000/versionExample annotation profilesThe following section provides a number of example annotation profiles illustrating the use of Inscriptis’ annotation support.
The examples present the used annotation rules and an image that highlights a snippet with the annotated text on the converted web page, which has been
created using the HTML postprocessor as outlined in Sectionannotation postprocessors.Wikipedia tables and table metadataThe following annotation rules extract tables from Wikipedia pages, and annotate table headings that are typically used to indicate column or row headings.{"table":["table"],"th":["tableheading"],"caption":["caption"]}The figure below outlines an example table from Wikipedia that has been annotated using these rules.References to entities, missing entities and citations from WikipediaThis profile extracts references to Wikipedia entities, missing entities and citations. Please note that the profile isn’t perfect, since it also annotates[ edit ]links.{"a#title":["entity"],"a#class=new":["missing"],"class=reference":["citation"]}The figure shows entities and citations that have been identified on a Wikipedia page using these rules.Posts and post metadata from the XDA developer forumThe annotation rules below, extract posts with metadata on the post’s time, user and the user’s job title from the XDA developer forum.{"article#class=message-body":["article"],"li#class=u-concealed":["time"],"#itemprop=name":["user-name"],"#itemprop=jobTitle":["user-title"]}The figure illustrates the annotated metadata on posts from the XDA developer forum.Code and metadata from Stackoverflow pagesThe rules below extracts code and metadata on users and comments from Stackoverflow pages.{"code":["code"],"#itemprop=dateCreated":["creation-date"],"#class=user-details":["user"],"#class=reputation-score":["reputation"],"#class=comment-date":["comment-date"],"#class=comment-copy":["comment-comment"]}Applying these rules to a Stackoverflow page on text extraction from HTML yields the following snippet:Advanced topicsAnnotated textInscriptis can provide annotations alongside the extracted text which allows
downstream components to draw upon semantics that have only been available in
the original HTML file.The extracted text and annotations can be exported in different formats,
including the popular JSONL format which is used bydoccano.Example output:{"text":"Chur\n\nChur is the capital and largest town of the Swiss canton
          of the Grisons and lies in the Grisonian Rhine Valley.","label":[[0,4,"heading"],[0,4,"h1"],[6,10,"emphasis"]]}The output above is produced, if inscriptis is run with the following
annotation rules:{"h1":["heading","h1"],"b":["emphasis"],}The code below demonstrates how inscriptis’ annotation capabilities can
be used within a program:importurllib.requestfrominscriptisimportget_annotated_text,ParserConfigurl="https://www.fhgr.ch"html=urllib.request.urlopen(url).read().decode('utf-8')rules={'h1':['heading','h1'],'h2':['heading','h2'],'b':['emphasis'],'table':['table']}output=get_annotated_text(html,ParserConfig(annotation_rules=rules)print("Text:",output['text'])print("Annotations:",output['label'])Fine tuningThe following options are available for fine tuning inscriptis’ HTML rendering:More rigorous indentation:callinscriptis.get_text()with the
parameterindentation='extended'to also use indentation for tags such as<div>and<span>that do not provide indentation in their standard
definition. This strategy is the default ininscriptand many other
tools such as Lynx. If you do not want extended indentation you can use the
parameterindentation='standard'instead.Overwriting the default CSS definition:inscriptis uses CSS definitions
that are maintained ininscriptis.css.CSSfor rendering HTML tags. You can
override these definitions (and therefore change the rendering) as outlined
below:fromlxml.htmlimportfromstringfrominscriptis.css_profilesimportCSS_PROFILES,HtmlElementfrominscriptis.html_propertiesimportDisplayfrominscriptis.model.configimportParserConfig# create a custom CSS based on the default style sheet and change the# rendering of `div` and `span` elementscss=CSS_PROFILES['strict'].copy()css['div']=HtmlElement(display=Display.block,padding=2)css['span']=HtmlElement(prefix=' ',suffix=' ')html_tree=fromstring(html)# create a parser using a custom cssconfig=ParserConfig(css=css)parser=Inscriptis(html_tree,config)text=parser.get_text()Custom HTML tag handlingIf the fine-tuning options discussed above are not sufficient, you may even override Inscriptis’ handling of start and end tags as outlined below:inscriptis=Inscriptis(html,config)inscriptis.start_tag_handler_dict['a']=my_handle_start_ainscriptis.end_tag_handler_dict['a']=my_handle_end_atext=inscriptis.get_text()In the example the standard HTML handlers for theatag are overwritten with custom versions (i.e.,my_handle_start_aandmy_handle_end_a).
You may define custom handlers for any tag, regardless of whether it already exists instart_tag_handler_dictorend_tag_handler_dict.Optimizing memory consumptionInscriptis uses the Python lxml library which prefers to reuse memory rather than release it to the operating system. This behavior might lead to an increased memory consumption, if you use inscriptis within a Web service that parses very complex HTML pages.The following code mitigates this problem on Unix systems by manually forcing lxml to release the allocated memory:importctypesdeftrim_memory()->int:libc=ctypes.CDLL("libc.so.6")returnlibc.malloc_trim(0)ExamplesStrict indentation handlingThe following example demonstrates modifyingParserConfigfor strict indentation handling.frominscriptisimportget_textfrominscriptis.css_profilesimportCSS_PROFILESfrominscriptis.model.configimportParserConfigconfig=ParserConfig(css=CSS_PROFILES['strict'].copy())text=get_text('fi<span>r</span>st',config)print(text)Ignore elements during parsingOverwriting the default CSS profile also allows changing the rendering of selected elements.
The snippet below, for example, removes forms from the parsed text by setting the definition of theformtag toDisplay.none.frominscriptisimportget_textfrominscriptis.css_profilesimportCSS_PROFILES,HtmlElementfrominscriptis.html_propertiesimportDisplayfrominscriptis.model.configimportParserConfig# create a custom CSS based on the default style sheet and change the# rendering of `div` and `span` elementscss=CSS_PROFILES['strict'].copy()css['form']=HtmlElement(display=Display.none)# create a parser configuration using a custom csshtml="""First line.
          <form>
            User data
            <label for="name">Name:</label><br>
            <input type="text" id="name" name="name"><br>
            <label for="pass">Password:</label><br>
            <input type="hidden" id="pass" name="pass">
          </form>"""config=ParserConfig(css=css)text=get_text(html,config)print(text)CitationThere is aJournal of Open Source Softwarepaperyou can cite for Inscriptis:@article{Weichselbraun2021,doi={10.21105/joss.03557},url={https://doi.org/10.21105/joss.03557},year={2021},publisher={The Open Journal},volume={6},number={66},pages={3557},author={Albert Weichselbraun},title={Inscriptis - A Python-based HTML to text conversion library optimized for knowledge extraction from the Web},journal={Journal of Open Source Software}}ChangelogA full list of changes can be found in therelease notes.