Added scripts to run the validation tests for each crawler.

Instead of running the suite of sites tests automatically when the other
unit tests are run scripts were added so they can be run separately. This
makes it easier to specify the parameters used to access the sites, by
passing them on the command line.
Split REQUIREMENTS.txt into separate files.

Split the requirements into separate files for development, staging and
production. This ensures that packages used for testing are not installed
into a production environment.
Crawlers are now configured using environment variables.

Instead of configuring the crawlers by overriding default values using a
local settings file, the main settings file now initializes the settings
from environment variables. This makes it a lot easier to install the
package since the configuration is now performed at runtime, rather than
install time (since the local setting file had to be created and the
values set when the package was installed). This also make it easier to
run the crawlers in the most common (for now) use-case where they are
called from a cron-like scheduler.

The names of the settings variables were shortened since - there was no
need to keep the "CHECKLISTING_" prefix for example. The settings that
defined the number of days to download checklists were configurable on
a per-crawler basis (using the settings EBIRD_DURATION and
WORLDBIRDS_DURATION) were replaced by a single setting, DURATION since
this can be changed at runtime if necessary.

The names of the environment variables match those of the settings with
the prefix "CHECKLISTING_" added, e.g. "CHECKLISTING_DOWNLOAD_DIR" so
there are no collisions with other environment variables.

Rewrote the documentation to reflect this change.