Package: checklists_scrapers
===================Checklists_scrapers===================Checklists_scrapers is a set of web scrapers for downloading records fromon-line databases of observations of birds. Scrapers are available for::ebird:a database hosted by the Laboratory for Ornithology at Cornell University,covering the Americas, Oceania and an increasing number of records forEuropean countries.:worldbirds:a network of databases hosted by WorldBirds (BirdLife International),with good coverage of countries around the Mediterranean and Africa.The eBird API only provides checklist records for up to the past 30 days sothe scrapers must be run on a regular basis. They are intended to provide acontinuous update of records and so are ideal for mirroring subsets of therecords available (for a given region for example) so you don't have torepeatedly run reports or submit requests for data from the database hosts.So, what is this for?---------------------Checklists_scrapers was written to aggregate records from different databasesfor loading into a`django-checklists <http://github.com/StuartMacKay/django-checklists>`_database. However, since the downloaded checklists are in JSON format the filemay be used with any similar database.The scrapers (and django-checklists) are current used by the`Birding Lisboa <http://www.birdinglisoa.com/>`_ news service which covers thearea around the Tejo estuary, Portugal. All the downloaded checklists areloaded into a database which is used to publish the latest news as well asgenerate annual reports.A similar database could be used for any purpose - analysing observationsfor conservation, environmental management or education. Aggregating theobservations from multiple databases with the scrapers makes this task alot easier.Installing & Configuring------------------------Checklists_scrapers is available from PyPI. You can install it with pip oreasy_install::pip install checklists_scrapersThe scrapers are built using the scrapy engine which uses settings, in the sameway as Django, for configuration and runtime values. The settings file isconfigured to initialize its values from environment variables. That makes iteasy to configure the scrapers, particularly for the most common use-case,running them from a scheduler such as cron.The only required setting is to tell scrapy (the engine used by the scrapers)the path to the settings module::export SCRAPY_SETTINGS_MODULE=checklists_scrapers.settingsThe remaining settings have sensible defaults so only those that areinstallation dependent, such as the mail server used for sending out statusreports. Here is this script that is used to run the scrapers for BirdingLisboa from cron::#!/bin/bashexport SCRAPY_SETTINGS_MODULE=checklists_scrapers.settingsexport LOG_LEVEL=INFOexport DOWNLOAD_DIR=/tmp/checklists_scrapersexport MAIL_HOST=mail.example.comexport MAIL_PORT=25export MAIL_USER=<user>export MAIL_PASS=<password>export MAIL_FROM=scrapers@example.comexport REPORT_RECIPIENTS=admins@example.comsource /home/project/venv/bin/activatecd /home/projectscrapy crawl ebird -a region=PT-11scrapy crawl ebird -a region=PT-15The settings can also be defined when the scrapers are run using the -Soption::scrapy crawl ebird -a region=PT-15 -s LOG_LEVEL=DEBUGHowever this obvious becomes a little cumbersome if more than one or twosettings are involved.scrapy crawl ebird -a region=PT-15 -s DOWNLOAD_DIR=downloadsNOTE: the environment variables use a prefix "CHECKLISTS" as a namespaceto avoid interfering with any other variables. When the setting is definedusing the -s option when running the scrapers, this prefix must be dropped::NOTE: REPORT_RECIPIENTS is a comma-separated list of one or moreemail addresses. The default value is an empty string so no status reportswill be mailed out. However if the LOG_LEVEL is set to 'DEBUG' the statusreport will be written to the file checklists_scrapers_status.txt in theDOWNLOAD_DIR directory.Everything is now ready to run.Scraping--------The arguments passed to the scrapers on the command line specify value such aswhich region to download observations from and login details for scrapersthat need an account to access the data::scrapy crawl ebird -a region=PT-11scrapy crawl worldbirds -a username=<user> -a password=<pass> -a country=ukSee the docs for each spider to get a list of the command line arguments andsettings.If you have defined the settings for a mail server and the settingREPORT_RECIPIENTS then a status report will be sent out each timethe scrapers are run. If the LOG_LEVEL is set to 'DEBUG' the report is alsowritten to the directory where the checklists are downloaded to. The reportcontains a list of the checklist downloaded along with an errors (complete withstack traces) and any warnings::Scraper: ebirdDate: 03 Jan 2014Time: 11:00-------------------------Checklists downloaded-------------------------2013-12-27 09:59, Jardim Botanico da Universidade de Lisboa2013-12-28 10:20, Baia Cascais2013-12-28 13:31, PN Sintra-Cascais--Cabo da Roca2013-12-29 07:45, RN Estuario do Tejo--Vala da Saragossa----------Errors----------URL: http://ebird.org/ebird/view/checklist?subID=S161101101Traceback (most recent call last):File "/home/birdinglisboa/venv/local/lib/python2.7/site-packages/twisted/internet/base.py", line 1201, in mainLoopself.runUntilCurrent()File "/home/birdinglisboa/venv/local/lib/python2.7/site-packages/twisted/internet/base.py", line 824, in runUntilCurrentcall.func(*call.args, **call.kw)File "/home/birdinglisboa/venv/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 382, in callbackself._startRunCallbacks(result)File "/home/birdinglisboa/venv/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 490, in _startRunCallbacksself._runCallbacks()--- <exception caught here> ---File "/home/birdinglisboa/venv/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 577, in _runCallbackscurrent.result = callback(current.result, *args, **kw)File "/home/birdinglisboa/venv/local/lib/python2.7/site-packages/checklists_scrapers/spiders/ebird_spider.py", line 585, in parse_checklistchecklist = self.merge_checklists(original, update)File "/home/birdinglisboa/venv/local/lib/python2.7/site-packages/checklists_scrapers/spiders/ebird_spider.py", line 602, in merge_checklistsoriginal['entries'], update['entries'])File "/home/birdinglisboa/venv/local/lib/python2.7/site-packages/checklists_scrapers/spiders/ebird_spider.py", line 695, in merge_entriesif count in key[index]:exceptions.TypeError: string indices must be integers------------Warnings------------2014-01-01 11:55, Parque da PazAPI: http://ebird.org/ws1.1/data/obs/loc/recent?r=L1127099&detail=full&back=7&includeProvisional=true&fmt=jsonURL: http://ebird.org/ebird/view/checklist?subID=S16160707Could not update record from API. There are 2 records that match: species=White Wagtail; count=4.Checklists downloaded also included the name of the observer, which was removedhere for obvious reasons. The stack traces in the Errors section is useful ifthere is a bug but it is also a first indication that the format of theinformation being scraped has changed. In either case report it as an issue andit will get fixed.Warnings are generally informative. Here a warning is generated because thechecklist contained two equal counts for White Wagtail in the API records -only the species is reported information on subspecies is dropped. Howeverthe subspecies is reported on the checklist web page. That means when the webpage was scraped it was not possible to distinguish between the two records.The records should be edited to add any useful information such as comments,which are only available from the web page.Links#####* Documentation: http://checklists_scrapers.readthedocs.org/* Repository: https://github.com/StuartMacKay/checklists_scrapers/* Package: https://pypi.python.org/pypi/checklists_scrapers/* Buildbot: http://travis-ci.org/#!/StuartMacKay/checklists_scrapers/.. image:: https://secure.travis-ci.org/StuartMacKay/checklists.png?branch=master:target: http://travis-ci.org/StuartMacKay/checklists_scrapers/Licence#######Checklists_scrapers is available under the modified BSD licence.