Package: doc_crawler
doc_crawler - explore a website recursively and download all the wanted documents (PDF, ODT…).== Synopsisdoc_crawler.py [--accept=jpe?g$] [--download] [--single-page] [--verbose] http://…doc_crawler.py [--wait=3] [--no-random-wait] --download-files url.lstdoc_crawler.py [--wait=0] --download-file http://…orpython3 -m doc_crawler […] http://…== Description_doc_crawler_ can explore a website recursively from a given URL and retrieve, in thedescendant pages, the encountered document files (by default: PDF, ODT, DOC, XLS, ZIP…)based on regular expression matching (typically against their extension). Documents can belisted on the standard output or downloaded (with the _--download_ argument).To address real life situations, activities can be logged (with _--verbose_). +Also, the search can be limited to one page (with the _--single-page_ argument).Documents can be downloaded from a given list of URL, that you may have previouslyproduced using default options of _doc_crawler_ and an output redirection such as: +`./doc_crawler.py http://… > url.lst`Documents can also be downloaded one by one if necessary (to finish the work), using the_--download-file_ argument, which makes _doc_crawler_ a tool sufficient by itself to assist youat every steps.By default, the program waits a randomly-pick amount of seconds, between 1 and 5, before eachdownload to avoid being rude toward the webserver it interacts with (and so avoid beingblack-listed). This behavior can be disabled (with a _--no-random-wait_ and/or a _--wait=0_argument)._doc_crawler.py_ works great with Tor : `torsocks doc_crawler.py http://…`== Options*--accept*=_jpe?g$_::Optional regular expression (case insensitive) to keep matching document names.Example : _--accept=jpe?g$_ will keep all : .JPG, .JPEG, .jpg, .jpeg*--download*::Directly downloads found documents if set, output their URL if not.*--single-page*::Limits the search for documents to download to the given URL.*--verbose*::Creates a log file to keep trace of what was done.*--wait*=x::Change the default waiting time before each download (page or document).Example : _--wait=3_ will wait between 1 and 3s before each download. Default is 5.*--no-random-wait*::Stops the random pick up of waiting times. _--wait=_ or default is used.*--download-files* url.lst::Downloads each documents which URL are listed in the given file.Example : _--download-files url.lst_*--download-file* http://…::Directly save in the current folder the URL-pointed document.== TestsAround 30 _doctests_ are included in _doc_crawler.py_. You can run them with the followingcommand in the cloned repository root: +`python3 -m doctest doc_crawler.py`Tests can also be launched one by one using the _--test=XXX_ argument: +`python3 -m doc_crawler --test=download_file`Tests are successfully passed if nothing is output.== Requirements- requests- yamlOne can install them under Debian using the following command : `apt install python3-requests python3-yaml`== AuthorSimon Descarpentries - https://s.d12s.fr== RessourcesGithub repository : https://github.com/Siltaar/doc_crawler.py +Pypi repository : https://pypi.python.org/pypi/doc_crawler== SupportTo support this project, you may consider (even a symbolic) donation via : https://liberapay.com/Siltaar== LicenceGNU General Public License v3.0. See LICENCE file for more information.