minor fixes in the crawl ordering and delay between requests