Scripts/reports: parse opts in parse-aafds-logs.pl

Add options parsing to parse-aafds-logs.pl.

Specifically, implement a --dry-run option (as already exists in
parse-ds-logs.pl) to do a dry run without inserting the records into the
database.
Scripts/reports: fix DB schema use for FR2 in parse-aafds-logs.pl

Bring parse-aafds-logs.pl up to date for FR2 - change SQL code to
reflect the DB schema changes done in FR2.
Scripts/reports: add scripts to pull remote logs

Add scripts for pulling remote logs (so far in AAFDS syntax).

* pull-aafds.conf has configuration, each line representing a remote location, with the fields being:
     REMOTE_BASE_URL USERNAME PASSWORD LOCALFILE_BASE_NAME
** E.g.:
    https://remote.host/logur/logfile.log user PASSW0RD /opt/shibboleth-ds/remote-logs/remote.host/logfile.log

* pull-aafds-logs.sh would fetch the URL (with "-$TODAY" appended with 'date +%Y%m%d' syntax), store in a local file (also with "-$TODAY" appended) and feed the contents to parse-aafds-logs.pl
** This filename syntax assumes logrotate renaming the file to TODAY (as done with the dateext option)

* Invoke pull-aafds-logs.sh from cron at a time AFTER logrotate.d is processed - check the remote system's /etc/crontab
Scripts/reports: minor exts to pull-aafds-logs.sh

Minor extensions to scripts/reports/shibbolethDS/pull-aafds-logs.sh
* allow passing "--dry-run" to be passed to parser script
* catch failures of wget and parser script and report (and delete incomplete download)
Scripts/reports: add DS host option to log parsers

Add an --ds-host <hostname> option to log parsers (parse-aafds-logs.pl
and parse-ds-logs.pl) to allow parsing logs from a remote system and
correctly store the hostname of the actual DS handling the session.
Scripts/reports: pull-aafds: remote DS host name

Add the hostname of the remote DS host to the configuration file
pull-aafds.conf and pass that to the log parser in pull-aafds-logs.sh
Scripts/reports: generic pulling of remote logs

Add a generic script for pulling remote logs (superceding the
specialized pull-aafds-logs.sh script)

Configuration is defined in pull-remote-logs.conf where each line consists of:
* LOG_ID - a short ID of the configuration entry (used to name a directory where the remote logs from this source will be stored)
* REMOTE_DS_HOST - the name of this DS host (to be passed to log parser scripts and stored in the database as ds_host)
* LOG_TYPE - currently either AAF-DS or Tuakiri-DS (selects which log parser to use - either parse-aafds-logs.pl or parse-ds-logs.pl)
* URL_BASE - the base URL where to fetch logs from (must support directory listing)
* USERNAME - username to use for authenticated access to the remote URL
* PASSWORD - corresponding password

The script pull-remote-logs.sh for each of the line in the configuration file:
- fetches the directory listing and parses it for file names (assuming an Apache-generated index page)
- for each file, checks whether the file already exists in the local directory
- if not, fetch the file and pass it to the local parser
Scripts/reporting: fix typo (URL) in sample conf

Fix typo in sample URL in pull-aafds.conf, the sample config file for
pull-aafds-logs.sh
Scripts/reporting: change base dir for remote logs

In pull-remote-logs.sh, change the base directory used for storing
remote logs - to "$DIR_BASE/../remote-logs", allowing to have the script
in "bin" directory and logs in a sibling directory.
Scripts/reports: remote log fetching - new version

A number of changes to pull-remote-logs.sh:

* changed config file format (shell script setting variables)
* functions for error reporting (and sending alert emails)
* error checking for remote fetching and parser invocations
Scripts/reports: add WGET_OPTIONS to log fecher

Add support for WGET_OPTIONS to pull-remote-logs.sh and
pull-remote-logs.conf (including "--quiet" in the default configuration).
Scripts/reporting: log fetcher: local wget options

More work on log fetcher (pull-remote-logs.sh)
* allow local (per source) wget options
* delete local log files if either running in dry-run mode or if parser
  failed (sticking to invariant that all locally stored log files have
  been successfully processed).
* minor fix (remove exit command from report_error function)
Scripts/reports: added remote log fetching doco

Added Remote-Log-Fetching.md
Scripts/reports: minor update to doco

Dropped extra pasted text from Remote-Log-Fetching.md
Scripts/reports: updated doco: reference Tukairi

As per comments by @braddleybedoes
Merge pull request #175 from University-of-Auckland/develop-DS-usage-scripts

DS usage scripts - remote log fetching