Add the ability to externalize the enforcement of the robots.txt Crawl-delay directive.