Add the ability to externalize the enforcement of the robots.txt Crawl-delay directive.
Merge pull request #6 from NearbyFYI/ignore_crawl_delay

Enable externalization of Crawl-delay directive enforcement from allowed?() and disallowed?().