Enable externalization of Crawl-delay directive enforcement from allowed?() and disallowed?().

https://github.com/knu/webrobots/issues/5