Max crawl delay

Pretty straightforward patch here. Just adds a MaxCrawlDelay to prevent robots.txt from sticking walker in the mud.