store crawl-delays of -1 so we at least know
we tried to get it from robots.txt and can
spider with the default 3 concurrent spiders.