Do not end a crawl while urls are still being spidered
because they might add more links to spiderdb when
they finally complete.