Enable externalization of Crawl-delay directive enforcement from allowed?() and disallowed?().

Consider the following use case. A web crawler downloads an HTML page from a site that has a robots.txt Crawl-delay directive and it wants to filter out links on the page that it is not allowed to crawl. Each call to disallowed?(url) (or allowed?(url) for that matter) will enforce the crawl delay. It would be much better to allow the web crawler to respect the crawl delay before it requests resources that it is allowed to access.