Do not end a crawl while urls are still being spidered
because they might add more links to spiderdb when
they finally complete.
Merge branch 'diffbot' into diffbot-testing