Add the ability to externalize the enforcement of the robots.txt Crawl-delay directive.
Merge branch 'ignore_crawl_delay' of https://github.com/NearbyFYI/webrobots into ignore_crawl_delay