Package: htmlq
htmlq - command line HTML queryThis is a simple command line utility to query HTML content as if you were using jQuery selectors. The idea is to be able to use commands like the next one:$wget-q-O-www.google.com|htmlqtitleand get an output like this one:<title>Google</title>The command is mainly useful for scripting. For example, it is possible to try to get the version of a wordpress installation, by checking themeta name="generator"tag:$htmlq-uwww.wordpress.com'meta[name="generator"]'<meta content="WordPress.com" name="generator"/>Or the title of a web page:$htmlq-uwww.google.comtitle<title>Google</title>It is even possible to get the value of a particular attribute in a tag$htmlq-uwww.wordpress.com'meta[name="generator"]'-anamegeneratorOr even do more sophisticated queries than remove internal elements, empty values, etc. As an example, the next query gets the title string of the different items in a search of items in ebay:$ htmlq -u "https://www.ebay.com/sch/i.html?_nkw=laptop"  "li.s-item h3.s-item__title" -s "\n" --rm span -a . -nInstallingInstall usingpython-pip:$ pip install htmlqor building from source:$ pip install bs4 html5lib urllib3 requests pathlib
...
$ git clone https://github.com/dealfonso/htmlq.git
$ cd htmlq
$ python3 setup.py installUse CasesThe most simple way to usehtmlqis to get a tag from a web page:$htmlq-uwww.github.comtitle<title>GitHub: Where the world builds software · GitHub</title>(*) this example gets the title of a web pageBut we may want to get other tag...$htmlq-uwww.github.coma<a class="px-2 py-4 color-bg-info-inverse color-text-white show-on-focus js-skip-to-content" href="#start-of-content">Skip to content</a><a href="https://docs.github.com/articles/supported-browsers">Learn more about the browsers we support.</a><a aria-label="Homepage" class="mr-4" data-ga-click="(Logged out) Header, go to homepage, icon:logo-wordmark" href="https://github.com/"><svg aria-hidden="true" class="octicon octicon-mark-github color-text-white" data-view-component="true" height="32" version="1.1" viewBox="0 0 16 16" width="32"><path d="M8 0C3.58 0 0 3.58 0 8c0 3.54 2.29 6.53 5.47 7.59.4.07.55-.17.55-.38 0-.19-.01-.82-.01-1.49-2.01.37-2.53-.49-2.69-.94-.09-.23-.48-.94-.82-1.13-.28-.15-.68-.52-.01-.53.63-.01 1.08.58 1.23.82.72 1.21 1.87.87 2.33.66.07-.52.28-.87.51-1.07-1.78-.2-3.64-.89-3.64-3.95 0-.87.31-1.59.82-2.15-.08-.2-.36-1.02.08-2.12 0 0 .67-.21 2.2.82.64-.18 1.32-.27 2-.27.68 0 1.36.09 2 .27 1.53-1.04 2.2-.82 2.2-.82.44 1.1.16 1.92.08 2.12.51.56.82 1.27.82 2.15 0 3.07-1.87 3.75-3.65 3.95.29.25.54.73.54 1.48 0 1.07-.01 1.93-.01 2.2 0 .21.15.46.55.38A8.013 8.013 0 0016 8c0-4.42-3.58-8-8-8z" fill-rule="evenodd"></path></svg></a><a class="d-inline-block d-lg-none f5 color-text-white no-underline border color-border-tertiary rounded-2 px-2 py-1 mr-3 mr-sm-5" data-hydro-click='{"event_type":"authentication.click","payload":{"location_in_page":"site header","repository_id":null,"auth_type":"SIGN_UP","originating_url":"https://github.com/","user_id":null}}' data-hydro-click-hmac="520d87e8f83281e6946b192f0f840552721c7fcba9b9c36d802e898a816314e2" href="/signup?ref_cta=Sign+up&amp;ref_loc=header+logged+out&amp;ref_page=%2F&amp;source=header-home">Sign up...(*) this example gets the "a" tags of a web pageWe obtained a lot of information, and that is why we wanted to narrow the query to remove those that we do not need$htmlq-uwww.github.com"a[aria-label]"<a aria-label="Homepage" class="mr-4" data-ga-click="(Logged out) Header, go to homepage, icon:logo-wordmark" href="https://github.com/"><svg aria-hidden="true" class="octicon octicon-mark-github color-text-white" data-view-component="true" height="32" version="1.1" viewBox="0 0 16 16" width="32"><path d="M8 0C3.58 0 0 3.58 0 8c0 3.54 2.29 6.53 5.47 7.59.4.07.55-.17.55-.38 0-.19-.01-.82-.01-1.49-2.01.37-2.53-.49-2.69-.94-.09-.23-.48-.94-.82-1.13-.28-.15-.68-.52-.01-.53.63-.01 1.08.58 1.23.82.72 1.21 1.87.87 2.33.66.07-.52.28-.87.51-1.07-1.78-.2-3.64-.89-3.64-3.95 0-.87.31-1.59.82-2.15-.08-.2-.36-1.02.08-2.12 0 0 .67-.21 2.2.82.64-.18 1.32-.27 2-.27.68 0 1.36.09 2 .27 1.53-1.04 2.2-.82 2.2-.82.44 1.1.16 1.92.08 2.12.51.56.82 1.27.82 2.15 0 3.07-1.87 3.75-3.65 3.95.29.25.54.73.54 1.48 0 1.07-.01 1.93-.01 2.2 0 .21.15.46.55.38A8.013 8.013 0 0016 8c0-4.42-3.58-8-8-8z" fill-rule="evenodd"></path></svg></a><a aria-label="Go to GitHub homepage" class="color-text-primary" data-hydro-click='{"event_type":"analytics.event","payload":{"category":"Footer","action":"go to home","label":"text:home","originating_url":"https://github.com/","user_id":null}}' data-hydro-click-hmac="062d687e04e8668f63bed700cfd9281766aa03a46d1beb6c750d751f692a1442" href="/"><img alt="GitHub" class="footer-logo-mktg" decoding="async" height="30" loading="lazy" src="https://github.githubassets.com/images/modules/site/icons/footer/github-logo.svg" width="84"/></a>(*) this example gets the link tags of a web page, but those that do have the attribute aria-label setBut we only want to select theatag, without some of the inner tags$htmlq-uwww.github.com"a[aria-label]"--rmsvg--rmimg<a aria-label="Homepage" class="mr-4" data-ga-click="(Logged out) Header, go to homepage, icon:logo-wordmark" href="https://github.com/"></a><a aria-label="Go to GitHub homepage" class="color-text-primary" data-hydro-click='{"event_type":"analytics.event","payload":{"category":"Footer","action":"go to home","label":"text:home","originating_url":"https://github.com/","user_id":null}}' data-hydro-click-hmac="062d687e04e8668f63bed700cfd9281766aa03a46d1beb6c750d751f692a1442" href="/"></a>(*) the --rm parameter enables to remove inner queries on each entryBut we only need the value of the label attribute, one on each line:$htmlq-uwww.github.com"a[aria-label]"--rmsvg--rmimg-aaria-label-s"\n"HomepageGo to GitHub homepage(*) the -a parameter enables to obtain the values of the attributes for each resulting entry, and -s sets the separator between resultsFinally we want to get also the destination URL, with a pretty arrow:htmlq -u www.github.com "a[aria-label]" --rm svg --rm img -a aria-label -a href -s "\n" -S " -> "Homepage -> https://github.com/Go to GitHub homepage -> /(*) we may include multiple attributes (using multiple -a entries) and join them with specific connectors using -SDetailed optionsThe usage syntax forhtmlqis the next:htmlq [-h] [-f FILENAME] [-u URL] [-a ATTRIBUTE] [-r RMQUERY] [-s SEPARATOR] [-S FIELDSEPARATOR] [-n] [-N] [-1] [-U USER_AGENT] queryThere are multiple options and flags forhtmlqand here we try to explain each of them.-f | --filename <filename>reads content of the filefilename. It is possible to use the whole path to the file (e.g./path/to/my/file) or use special paths (e.g.~/myfile). If no filename nor url is included,htmlqwill read from the standard input.-u | --url <url>retrieves the content to be parsed from the url. It is advisable to include the whole schema in the url (e.g.https://my.url). If not included, thehttpsschema will be tried in first place, and if it fails,httpwill be tried. If a file is included in the commandline, this parameter will be ignored. If no filename nor url is included,htmlqwill read from the standard input.-a | --attr <attribute list>if a query obtains a set of tags as a result, the default behavior (if this parameter is not set) is to output the result of the whole obtained html fragments. Instead, if an attribute is queried (using-a) the output will be the values of each of these attributes for the entry. In case that an attribute is not in the html node, its output will beblank. It is possible to query multiple attributes by including multiple-aentries (e.g.-a href -a aria-label). There is an special attribute (.) which refers to the text representation of the entry (i.e.-a .).-r | --rm <query>the query to the html document may contain child nodes (e.g. <ul><li><li></ul>). When querying for<ul>, the result will be the<ul>node along with its<li>child nodes. Using-rit is possible to delete the<li>nodes. It is possible to remove multiple child trees by including multiple--rmqueries.-s | --separator <separator>this is the string used to join the output of the result of the different entries. It is possible to include escaped strings (e.g.\n) or whole arbitraty strings (e.g.\n ->). The default value is\0.-S | --field-separator <separator>this is the string used to join the values of the different attributes obtained from an entry, using-aparameter. It is possible to include escaped strings (e.g.\n) or whole arbitraty strings (e.g.->). The default value is,.-n | --no-empty-linesusing this flag,htmlqwill not include empty lines (i.e. lines whose value isblankas a result of the combination of attributes).-N | --no-empty-attrusing this flag,htmlqwill not include the value of attributes that are empty (i.e. lines whose value isblankas a result of the combination of attributes). Using this option, the number of resulting attributes may differ from the number of requested attributes (e.g.-a href -a class -a idmay be converted to/,mylinkifclassis not set for an entry).-1 | --only-firstif a query obtains multiple results, using this flag,htmlqwill deal only with the first one (thus ignoring the rest).-U | --user-agent <user agent string>. Using this parameter, it is possible to set an arbitraty user agent string to retrieve the web page. You can check your user agent string in this web:https://www.whatsmyua.infoquery. This is the query string that wants to be retrieved from the html web page. It is possible to use queries that retrieve multiple trees. In this case,htmlqwill consider them as individual entries and will deal with all of them (or only the first if using flag-1).urlf - format urlThis command is an add-on tohtmlq, as a command line application to deal with URLs and extracting information about them.The original purpose was to extract the values of variables in URLs, so that their values can be used in scripts. An example:$urlf-voq"https://www.google.com/search?q=github&oq=github&sourceid=chrome&ie=UTF-8"github(*) This example gets the value of varoq.Then the application has evolved to enable rewritting URLs, using the commandline as in the next example:$urlf"https://www.google.com/search\?q=github&oq=github&sourceid=chrome&ie=UTF-8"-F"%s://%H?oq=%#oq#"https://www.google.com?oq=github(*) This example rewrites the URL to build a new one that removes the path and just includes the value of varoq.Detailed optionsThere are multiple options and flags forurlfand here we try to explain each of them.usage: urlf [-h] [-U] [-s] [-u] [-w] [-H] [-p] [-P] [-q] [-m] [-f] [-v var name] [-j SEPARATOR] [-F format string] [-V] urls [urls ...]-h, --help: shows the help-V, --version: show program's version number and exit-U, --url: displays the URL as provided in the input.-s, --scheme: shows the scheme provided in the url (e.g. https)-u, --username: shows the username to accede to the url (i.e. user inhttps://user@pass:myserver.com)-w, --password: shows the password to accede to the url (i.e. pass inhttps://user@pass:myserver.com)-H, --hostname: shows the hostname in the url (i.e. myserver.com inhttps://myserver.com/my/path)-p, --port: shows the port in the url (i.e. 443 inhttps://myserver.com:443/my/path)-P, --path: shows the path in the url (i.e. my/path inhttps://myserver.com/my/path)-q, --query: shows the query in the url (i.e. q=1&r=2 inhttps://myserver.com/my/path?q=1&r=2)-m, --parameters: shows the parameters to accede to the url (i.e. param inhttps://myserver.com/my/path;param?q=1&r=2)-f, --fragment: shows the fragment in the url (i.e. sec1 inhttps://myserver.com/my/path#sec1)__-v var name, --var var name
show the value of a var in the query string (this parameter may appear multiple times, to get the values of multiple variables; they will appear in the same order than appeared in the commandline)-j | --join-string <SEPARATOR>:
character (or string) used to separate the different fields (default: )-F | --format-string <format string>:
user defined format string to get a custom output of the URL parts. Any arbitrary field or character may appear in this string, and the fields are substituted using the letter in the shorthand flag of each parameter, preceded by symbol %. E.g.urlf -His the same thanurlf -F "%H"; e.g.urlf -s -His the same thanurlf -F "%s%H", but you can useurlf -F "%s://%H"to obtain a better output. In the case of variables, the value is obtained by surrounding the name of the var by symbol # and prepending symbol %; e.g.urlf -v qis the same thanurlf -F "%#q#".A combined example (guessing wordpress version)In case that we wanted to get the version of a wordpress installation, we could check meta tag and get the content:$ htmlq -u www.grycap.upv.es 'meta[name="generator"]'
<meta content="WordPress 5.8.1" name="generator"/>
$ htmlq -u www.grycap.upv.es 'meta[name="generator"]' -a content
WordPress 5.8.1But many plugins hide the version in the tag, so we can try to guess the version from the links:$ htmlq -u www.grycap.upv.es 'link[href*="?ver="]' -s '\n'
<link href="https://www.grycap.upv.es/wp-includes/css/dist/block-library/style.min.css?ver=5.8.1" id="wp-block-library-css" media="all" rel="stylesheet" type="text/css"/>
<link href="https://www.grycap.upv.es/wp-content/themes/specia/style.css?ver=5.8.1" id="specia-style-css" media="all" rel="stylesheet" type="text/css"/>
<link href="https://www.grycap.upv.es/wp-content/themes/specia/css/colors/default.css?ver=5.8.1" id="specia-default-css" media="all" rel="stylesheet" type="text/css"/>
<link href="https://www.grycap.upv.es/wp-content/themes/specia/css/owl.carousel.css?ver=5.8.1" id="owl-carousel-css" media="all" rel="stylesheet" type="text/css"/>
<link href="https://www.grycap.upv.es/wp-content/themes/specia/css/bootstrap.css?ver=5.8.1" id="bootstrap-css" media="all" rel="stylesheet" type="text/css"/>
<link href="https://www.grycap.upv.es/wp-content/themes/specia/css/woo.css?ver=5.8.1" id="woo-css" media="all" rel="stylesheet" type="text/css"/>
<link href="https://www.grycap.upv.es/wp-content/themes/specia/css/form.css?ver=5.8.1" id="specia-form-css" media="all" rel="stylesheet" type="text/css"/>
<link href="https://www.grycap.upv.es/wp-content/themes/specia/css/typography.css?ver=5.8.1" id="specia-typography-css" media="all" rel="stylesheet" type="text/css"/>
<link href="https://www.grycap.upv.es/wp-content/themes/specia/css/media-query.css?ver=5.8.1" id="specia-media-query-css" media="all" rel="stylesheet" type="text/css"/>
<link href="https://www.grycap.upv.es/wp-content/themes/specia/css/widget.css?ver=5.8.1" id="specia-widget-css" media="all" rel="stylesheet" type="text/css"/>
<link href="https://www.grycap.upv.es/wp-content/themes/specia/css/animate.min.css?ver=5.8.1" id="animate-css" media="all" rel="stylesheet" type="text/css"/>
<link href="https://www.grycap.upv.es/wp-content/themes/specia/css/text-rotator.css?ver=5.8.1" id="specia-text-rotator-css" media="all" rel="stylesheet" type="text/css"/>
<link href="https://www.grycap.upv.es/wp-content/themes/specia/css/menus.css?ver=5.8.1" id="specia-menus-css" media="all" rel="stylesheet" type="text/css"/>
<link href="https://www.grycap.upv.es/wp-content/themes/specia/inc/fonts/font-awesome/css/font-awesome.min.css?ver=5.8.1" id="font-awesome-css" media="all" rel="stylesheet" type="text/css"/>
<link href="https://www.grycap.upv.es/wp-content/plugins/forget-about-shortcode-buttons/public/css/button-styles.css?ver=2.1.2" id="forget-about-shortcode-buttons-css" media="all" rel="stylesheet" type="text/css"/>From the links, we see that wordpress includes the version of wordpress in the "ver" variable for any link; so we may get the value of such variable usingurlf:$ htmlq -u www.grycap.upv.es 'link[href*="?ver="]' -s '\n' -a href | ./urlf.py -v ver -
5.8.1
5.8.1
5.8.1
5.8.1
5.8.1
5.8.1
5.8.1
5.8.1
5.8.1
5.8.1
5.8.1
5.8.1
5.8.1
5.8.1
2.1.2
5.8.1And now, if we get the most used value, it will probably be the one that refers to the wordpress version (because other plugins may also use that variable for its purposes):$ htmlq -u www.grycap.upv.es 'link[href*="?ver="]' -s '\n' -a href | ./urlf.py -v ver - | sort | uniq -c | sort -k1 -n -r
  15 5.8.1
   1 2.1.2And the first one will be the most voted version.Now we can compare to the currently available wordpress version:$ curl -s https://api.wordpress.org/core/version-check/1.7/ | jq ".offers[].version" | tr -d '"' | sort -V | tail -n 1
5.8.1Our final script would be something like the next one:#!/bin/bashMYVERSION="$(htmlq-uwww.grycap.upv.es'link[href*="?ver="]'-s'\n'-ahref|./urlf.py-vver-|sort|uniq-c|sort-k1-n-r|head-n1|awk'{print $2}')"CURRENTVERSION="$(curl-shttps://api.wordpress.org/core/version-check/1.7/|jq".offers[].version"|tr-d'"'|sort-V|tail-n1)"LATESTVERSION="$(echo"$MYVERSION$CURRENTVERSION"|sort-V-r|head-n1)"if["$LATESTVERSION"="$MYVERSION"];thenecho"you have the latest version of wordpress ($LATESTVERSION)"elseecho"you should update your wordpress version"exit1fiexit0