Require Sidekiq debug module based on environment

Instead of commenting in and out the requirements for debugging the
Sidekiq procedures I have added a check whether the DEBUG environment
variable was set.

Signed-off-by: Konrad Reiche <konrad.reiche@gmail.com>
Optimize CKAN harvesting

By using the search endpoint as described in issue #1 the CKAN
harvesting performance was dramatically improved. For instance,
harvesting catalog.data.gov took something around 48 hours. Now
it takes something between 3 and 6 hours. This closes #1.

Signed-off-by: Konrad Reiche <konrad.reiche@gmail.com>
Add exception handling in case of blocked request

If there is a problem with the response, for instance I get forwarded to
a HTML page, then there will be a parsing error from the JSON module. I
have added exception handling which logs the occurrence and retries
after a couple of seconds. With every retry the sleep duration is
increased.

Signed-off-by: Konrad Reiche <konrad.reiche@gmail.com>
Add option to generate JSON dumps

For better archiving purposes and long-term use to run the harvesting
procedure as a cronjob on a server I have added an option to dump the
harvested data in their raw format. This closes #2.

Signed-off-by: Konrad Reiche <konrad.reiche@gmail.com>