Don't assign out further url discovery tasks after getting results if crawl job is still going on