a few hacks. temp hack for oom  to split 4 ways
for custom crawls
fix diffbot reply deduping to exclude url: field in the json
when doing deduping.
Merge branch 'diffbot-matt' of github.com:gigablast/open-source-search-engine into diffbot-matt
Merge branch 'diffbot-testing' into diffbot-matt
take out swap out stuff