GSA doesn't like null values in data.json files (revise output and amend validator)
the references field is an array, store it space-separated like we do with programCode and bureauCode, and on generating data.json files do a split on spaces
granularity is no longer in the POD schema since 1.0
add new 'overrides' harvester config, and refactor/simplify the data.json import code
add a new config option to exclude datasets by exact values of fields or regular expressions
dont put the URL of Query Tools in the webService field
GSA doesn't like empty lists in data.json files either (revise output and amend validator)
tighten the data.json validator to require that in ISO8601 fields the date/time separator be a T and not any character
fixes for generating the date fields (issued/modified/temporal): the date/time separator must be T (ISO 8601) and not a space (Drupal), when issued/modified looks like a plain date remove the midnight time component ('T00:00:00'), don't generate temporal if we only have a Coverage Period Fiscal Year Start/End and not regular dates or if either the start or end is not set (previously we generated 'Unknown' in the output which was definitely not valid); and update the validator to include the text of invalid dates in the output
code cleanup: simplifying HarvesterBase.load_config
move the logic of harvester defaults and overrides into harvester_base.py
now that package extras may be non-empty by the time we set its data (and overrides), check for whether an extra exists before appending an extra for a given key (see 85392f45c57d543bd8c4ff1ab09b7b73dea4bcd2)
the logic for putting lists and booleans into extras fields has to move to BaseHarvester too since that's where defaults are processed now
add a new configuraiton option ckanext.datajson.default_keywords for providing a 'keyword' value when a dataset has none
update README for how defaults/overrides work w.r.t. what the keys should be
default the 'modified' date to the 'issued' date
don't replace the default title/notes/author/url specified in a harvester config and not present in the data.json file
add a timeout to reading remote data.json files since some (SAMHSA) are hanging indefinitely
validator: dont warn about optional parameters in MIME types like '; charset=UTF-8' which I suppose are valid for the POD schema
put the count of datasets into the validator output