re-enable spidering if off because we missed
a crawlinfo reply from udp timed out.
Merge branch 'diffbot-testing' of github.com:gigablast/open-source-search-engine into diffbot-testing