Max crawl delay

Pretty straightforward patch here. Just adds a MaxCrawlDelay to prevent robots.txt from sticking walker in the mud.
Won't this fail to set the crawl delay if they set it lower than our default?