Package: heq
heq: Yet Another 'jq for HTML'heqis a command-line tool for extracting structured data as JSON from HTML using concise expressions, akin tojq. Additionally, heq serves as a Python library, facilitating the efficient scraping of HTML content through its jq-inspired DSL based on XPath.Installationpipinstallheqheq depends on the following Python packages:lxmlparsimoniousUsage as a command-line tool$cat<<'EOF'|heq'$`div.product` / {name: $`h2.name`.text}'<body><div id="header">Welcome to Our Store!</div><div class="product"><h2 class="name">Widget A</h2><p class="price">$10</p><ul class="features"><li>Durable</li><li>Lightweight</li></ul><a href="/products/widget_a">Details</a></div><div class="product"><h2 class="name">Gadget B</h2><p class="price">$20</p><ul class="features"><li>Compact</li><li>Energy Efficient</li></ul><a href="/products/gadget_b">Details</a></div></body>EOFOutput:[{"name":"Widget A"},{"name":"Gadget B"}]$catexpr.heq`//div[@class="product"]` / {name: `.//h2[@class="name"]`.text,price: `.//p[@class="price"]`.text,features: `.//li` / text,url: `.//a`@href}$cat<<'EOF'|heq-fexpr.heq(The same HTML as above)EOFOutput:[{"name":"Widget A","price":"$10","features":["Durable","Lightweight"],"url":"/products/widget_a"},{"name":"Gadget B","price":"$20","features":["Compact","Energy Efficient"],"url":"/products/gadget_b"}]Usage as a libraryfromheqimportextract,xpathhtml='''<body><div class="product"><h2 class="name">Widget A</h2><p class="price">$10</p><ul class="features"><li>Durable</li><li>Lightweight</li></ul></div><div class="product"><h2 class="name">Gadget B</h2><p class="price">$20</p><ul class="features"><li>Compact</li><li>Energy Efficient</li></ul></div></body>'''expr=xpath("//div[@class='product']")/{'name':xpath(".//h2[@class='name']").text,'price':xpath(".//p[@class='price']").text,'features':xpath(".//li")/{'feature':xpath('.').text}}print(extract(expr,html))Output:[{'name': 'Widget A',
  'price': '$10',
  'features': [{'feature': 'Durable'}, {'feature': 'Lightweight'}]},
 {'name': 'Gadget B',
  'price': '$20',
  'features': [{'feature': 'Compact'}, {'feature': 'Energy Efficient'}]}]Syntax and SemanticsInformal BNF-like Representation<S> ::= <expr>
<expr> ::= <selector_lit> '/' <term>
         | <term>
<term> ::= <dict_lit> | <dottext> | <atattr> | <filter>
<filter> ::= 'text' | <attr_lit>
<dict_lit> ::= '{' ((<dict_field_value> ',')* <dict_field_value>)? '}'
<dict_field_value> ::= <dict_field> ':' <expr>
<dottext> ::= <selector_lit> '.text'
<atattr> ::= <selector_lit> <attr_lit>
<selector_lit> ::= <css_lit> / <xpath_lit>
<css_lit> ::= '$' <backtick_lit>
<xpath_lit> ::= <backtick_lit>
<attr_lit> ::= '@' <ident_with_hyphen>heq has the concept ofcontext DOM tree. This is the DOM tree against which XPath expressions or CSS selectors are evaluated. Initially, it is set to the root tree, and it changes as the/operator is applied, to each of the elements.Available syntactic constructs and their semantics are as follows:Value Forms{key: expression}: Evaluates to a dictionary.keyis a string without quotes andexpressionis an expression.text: Evaluates to a string representing the text content of the context DOM tree.@attr: Evaluates to the value associated with the attributeattrof the context DOM tree.<selector>.text: Evaluates to a string representing the text content of the element(s) selected by the specified selector.<selector>@attr: Evaluates to a string representing the value associated with the attributeattrof the first element selected by the specified XPath expression.Selectors`<xpath>`: Selects elements by evaluating the XPath against the context DOM tree.$`<css_selector>`: Selects elements by evaluating the CSS selector against the context DOM tree.Mapping Against Query Results<selector> / <value_form>: First, evaluates the selector to obtain a list of elements. Then, for each element, thevalue_formis evaluated with the element as the new context DOM tree. The entire expression evaluates to an array.ExamplesTarget HTML<body><divid="header">Welcome to Our Store!</div><divclass="product"><h2class="name">Widget A</h2><pclass="price">$10</p><ulclass="features"><li>Durable</li><li>Lightweight</li></ul><ahref="/products/widget_a">Details</a></div><divclass="product"><h2class="name">Gadget B</h2><pclass="price">$20</p><ulclass="features"><li>Compact</li><li>Energy Efficient</li></ul><ahref="/products/gadget_b">Details</a></div></body>Example 1{ header: `//div[@id="header"]`.text }evaluates to:{"header":"Welcome to Our Store!"}Example 2`//div[@id="header"]`.textevaluates to:"Welcome to Our Store!"Example 3`//div[@class="product"]` / `.//a`@hrefevaluates to:["/products/widget_a","/products/gadget_b"]Example 4`//div[@class="product"]` / {
    name: `.//h2[@class="name"]`.text,
    price: `.//p[@class="price"]`.text,
    features: `.//li` / text,
    url: `.//a`@href
}evaluates to:[{"name":"Widget A","price":"$10","features":["Durable","Lightweight"],"url":"/products/widget_a"},{"name":"Gadget B","price":"$20","features":["Compact","Energy Efficient"],"url":"/products/gadget_b"}]Example 5$`div.product` / {
    name: $`h2.name`.text,
    price: $`p.price`.text,
    features: $`li` / text,
    url: $`a`@href
}evaluates to:[{"name":"Widget A","price":"$10","features":["Durable","Lightweight"],"url":"/products/widget_a"},{"name":"Gadget B","price":"$20","features":["Compact","Energy Efficient"],"url":"/products/gadget_b"}]