iParadigms/walker — pull request #75 by pchristopher1275 (2014-11-11 18:31:45+00:00)
Title: Max crawl delay

Pretty straightforward patch here. Just adds a MaxCrawlDelay to prevent robots.txt from sticking walker in the mud.