I prefer download a bunch of requests and determine the crawl time with CloseSpider Extension. We can do it by time, errors, items, pages or leave the ext disabled and relay on the HS_MAX_LINKS.