Package: web-trawler
Given the url of an html web page, this Python package asynchronously downloads all non-web
files linked to from that web page, e.g. audio files, Excel documents, etc. Optionally, all
web pages linked to from the original web page can be trawled for files as well.InstallationPython 3must be installed and in your system PATH. That is, it must be a recognised command
for the command line interface. Enterpython --versionin your command line to see whether
you have Python 3 installed.The Python package managerpipis also required. Check that you have it by runningpip --version.
It is automatically installed with recent versions of Python, but it can also be installed manually.
Seethe official installation instructionsInstalling the web_trawler packageRun the following code in your command line interface (excluding the $, which is just a prompt icon):$pipinstallweb_trawler--upgradeThe package has no external dependencies. For testing,pytestis required.The source code for web_trawler is available ongitlab.com.UsageCommand lineOnce installed, web_trawler can be used like this:$web_trawlergoogle.comRun this command to see how web_trawler finds links
and inspects their http headers for more information. A bunch of logging events will be output to console.
There are ordinarily no files linked to from google.com,
but if there are, they will be downloaded to the directorydownload/relative to where you ran the command.The url argument is required. In addition, the following optional arguments are supported:--targetTARGETGive a path for where you would like the files to be downloaded.
The default path is “download”.--add_links_from_linked_pagesSet web_trawler to trawl pages linked to from the original web page as well
(only goes one step,
and only for links within the domain of the original web page)--interactiveShort version is “-i”. Asks user about whether or not to trawl each linked page (has no effect
unless the –add_links_from_linked_pages flag is set to true.--interactive_download_promptShort version is “-I”. Asks user about whether or not to download each of the files found.--quietSuppresses output information about which links are being processed
and which files are being downloaded.--processesPROCESSESManually set how many processes will be spawned. The default is to spawn
one less than the number of processors detected (so as not to stall the
system). For each process, up to 10 threads are spawned.--whitelistWHITELISTSpace-separated file endings to whitelist. Allows use of wildcards, e.g.
“xls*” to capture all the Excel file extension variants, like xlsx, xlsb,
xlsm and xls. A given blacklist takes precedence over the whitelist.--blacklistBLACKLISTSpace-separated file endings to blacklist. Works just whitelist, only it
excludes files of the given file endings.--no_of_files_limitLIMITSet a maximum number of files you are willing to download, in case
web_trawler finds more than expected.--mb_per_file_limitLIMITSet a maximum file size you are willing to download. Warnings are
logged to console for each file excluded.Each argument has a shorthand consisting of their first letters, e.g.-t,-a,-q, etc.A realistic example of useIf we’d like to download, say, all zip and Excel files up to 100 MB froma web page on the World Input-Output Database site, into a local directory called “data”,
we’d need to use the arguments-t(for target),-w(for whitelist) and-m(for mb_per_file_limit):$web_trawlerhttp://www.wiod.org/database/wiots16-t"data"-w"zip xls*"-m100Notice the use of a wildcard in the whitelist. The web page specified links to two different Excel associated
file endings. The wildcard ensures that both are captured.If you test this command, downloads of a bunch of large files will start. Pressctrl-corctrl-zto
interrupt or force quit the process, respectively.Make sure to clean up any downloaded files you don’t want. They should be in a folder relative to where you ran the
command. If you didn’t specify a target, they are downloaded to a directory called “download”.Including links from linked pagesTo see how the-aargument works without starting a million downloads, run the following command, where-m 0ensures that all files are skipped:$web_trawlerhttp://www.wiod.org/database/wiots16-a-m0Note that this will still create the target directory if it doesn’t exist already.To get prompted about whether or not to add the files linked to from each of the linked pages, run this command,
where the-aand-icommands have been concatenated into one, and the whitelist is set so as to
not have any downloads started:$web_trawlerhttp://www.wiod.org/database/wiots16-ai-w"nosuchfileending"Use within PythonThe following code does the exact same thing as the last example for the command line usage:import web_trawler

web_trawler.trawl("http://www.wiod.org/database/wiots16",
                  add_links_from_linked_pages=True, mb_per_file_limit=0)The functiontrawldoes the same thing as web_trawler as run from the command line, but with the arguments
passed to it directly in Python.Several of the intermediary functions used in web_trawler can also be accessed through Python, i.e. to get a
list with information about all links on a webpage, or just the links to files, filtered with a blacklist
or whitelist. Here’s a brief description of each of them:get_links:Takes only one argument, a url, and returns a list of Link namedtuples, described below.
This list is unfiltered. All http links that return a http request are included.get_file_links:Runs get_links and returns a filtered list of Link namedtuples for files only,
with whitelist and/or blacklist applied if specified. Arguments have self-explanatory names.
The whitelist and blacklist can be provided as a space-separated string or as a list.Bothget_linksandget_file_linksreturn lists of namedtuples with the following fields:href:the link urltitle:the content of the<a>tag containing the linkmb:calculated from the http headercontent-lengthtype:the http headercontent-type, unmodifiedUse in MatlabIn Matlab, functions of pip installed Python packages can be called using thepyscript, where optional
arguments are specified using the pyargs function:>>py.web_trawler.get_file_links('http://www.wiod.org/database/wiots16',pyargs('whitelist','xls* doc*'))Stdout isn’t displayed, that’s why theget_file_linksfunction was chosen, as it returns something.
To use the full functionality of web_trawler, you could run the functiontrawlinstead. As long as
there are no errors, nothing will show up in the Command Window. Files will nevertheless be downloaded,
relative to your Current Folder in Matlab.