Max crawl delay

Pretty straightforward patch here. Just adds a MaxCrawlDelay to prevent robots.txt from sticking walker in the mud.
Aha you're right, not sure why I originally did it that way... I am thinking we probably want to let ourselves go faster for sites that deliberately give a low crawl delay though. Unless you disagree maybe we can change it to allow a crawl delay lower than default.