# Indexing Web with Head-r

Head-r is a free Perl program that recursively follows links located at (HTML) Web pages hosted on an HTTP server, and performs HEAD requests upon links of interest to the user.
The intended use for this program is to create URI lists for later selective mirroring of file-hosting sites.

## Basic usage

Arguably, the most important Head-r options are --info-re= and --descend-re=, which determine (by means of regular expressions) which URIs will be considered for mere HEAD requests, and which ones Head-r will try to get more URIs from.

### Simplistic, no-recursion example

For the following example, we’ll use . – a regular expression that matches any non-empty string – to allow Head-r to make HEAD requests to both of the URIs given.
The fields are delimited with ASCII HT (also known as TAB) codes, and are as follows:
* URI;
* timestamp (in seconds since system-dependent epoch; see also Unix time);
* recursion depth used when considering this URI;
* the length of the response in octets (as per the Content-Length: HTTP reply header);
* HTTP status code of the reply.

### Recurse once example

For the following example, we’ll also enable actual recursion (still at maximum depth of 1), by using the --descend-re=/\$ option.
As could be seen, at http://example.org/ Head-r found another URI to consider: http://www.iana.org/domains/example, which it followed and issued a HEAD request for.
It’s easy to check that http://example.net/ actually also references the same URI. However, as Head-r remembers the URIs it processes (along with the recursion depth at the point) no other request was issued.

### Limiting HEAD requests

Consider now that the resource we’re to recurse through references URIs that are out of our interest. For the following example, we’ll use a more selective regular expression than . we’ve used above.
(Please note that we’ve just used the Bash {,} expansion to pass the same regular expression to both --info-re= and --descend-re=. Be sure to adjust to the command line interpreter actually in use.)
In the output above, a number of URIs came without any of the usual information. These URIs were found by Head-r, but as they matched neither “info” (--info-re=) nor “descend” (--descend-re=) regular expressions specified, no action was done to them. The URIs are still output, however, just in case we may decide to adjust the regular expressions themselves.

### Skipping unwanted URIs altogether

The --include-re= and --exclude-re= regular expressions are considered before all the other ones, and currently have the following semantics:
* the inclusion regular expression is applied first; the URI will be considered if it matches one;
* unless decided at the step above, the exclusion regular expression is then applied; the URI will not be considered if it matches one;
* unless decided by the rules above, the URI will be considered.
If none of these options are given, any URI will be considered by Head-r.
The following example exploits these options to further limit the output of Head-r for the case above.

## Saving state between sessions

Head-r is capable of reading its own output, so to avoid issuing duplicate HEAD requests, and also to discover the URIs of the resources to recurse into.

### Restoring what was saved

Let us revisit one of our previous examples, which we’ll now alter to only issue a HEAD request to a couple of pages:
Now, why not to include a few more pages, such as all the pages with the names starting with F?
Note that while our --info-re= has obviously covered http://en.wikipedia.org/wiki/Free_content, no HEAD request was made to the page, as our --input=state.a file already had the relevant information.
Also, as all the URIs we wanted for Head-r to consider were already listed in state.a, it was unnecessary to specify any URIs at the command line. When the URIs come from both command line arguments and --input= files, those coming from command line are considered first.

### Compression

As recursing through large Web sites may result in large output lists, Head-r provides support for compression of output data.
The --bzip2 (-j) and --gzip (-z) options select the compression method to use for the output file (either specified with --output=, or standard output.) Head-r, however, will exit with an error if compression is enabled and the output goes to a terminal device.
Head-r transparently decompresses the files given as inputs (--input=), thanks to the IO::Uncompress::AnyUncompress library.

## Adjusting HTTP client behavior

There’re two options which influence the behavior of the HTTP client used by Head-r: --wait= (-w) and --user-agent= (-U.)
The --wait= option specifies the amount of time, in seconds, to wait between two consecutive HTTP requests. The default is about 2.7 seconds.
The --user-agent= option specifies the value for the User-Agent: header to use in HTTP requests, and may come handy should the target server block access based on this header’s data. The default is composed of the string HEAD-R-Bot/, the Head-r’s own version, and the identity of the libwww-perl library used. For example: HEAD-R-Bot/0.1 libwww-perl/6.05.

## Bugs

Please consider reporting any bugs in the Head-r software not listed below via the CPAN RT, https://rt.cpan.org/Public/Dist/Display.html?Name=head-r. The bugs in this documentation should be reported to the respective Wikibooks Talk page – or you may actually fix them yourself!
As for any other automatic retrieval tool, it isn’t impossible to abuse Head-r to cause excessive load on third-party servers. The user is advised to consider the network environment when using the tool, and especially when lowering the --wait= setting, and raising the maximum recursion --depth= beyond reasonable values.
There’s currently no way to disable the /robots.txt file processing.
The code only tries to retrieve URIs from content marked with text/html media type, even though it seems as if the support for application/xhtml+xml (and perhaps several other XML-based types, such as SVG) could be implemented rather easily.
The resource to retrieve URIs from is first loaded into memory, while it should be possible to process it on-the-fly.
The handling of recursion depths retrieved from --input= files may be somewhat unintuitive, and out of the user’s control. (Although it’s still possible to edit such files using third-party tools, such as AWK.)
The code implements a trivial work-around for the long-standing Net::HTTP bug #29468.

## Availability

The latest stable version of the code is available from CPAN. Check, for instance, the respective Metacpan page at https://metacpan.org/release/head-r.
The latest development version could be downloaded from a Git repository, like:
A Gitweb interface is available at http://am-1.org/~ivan/archives/git/gitweb.cgi?p=head-r-2013.git.

## Author

Head-r is written by Ivan Shmakov.
Head-r is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
This documentation is a free collaborative project going on at Wikibooks, and is available under the Creative Commons Attribution/Share-Alike License (CC BY-SA) version 3.0.