Add a :ignore_crawl_delay option that disables enforcement of the robots.txt Crawl-delay directive by allowed?(url) and disallowed?(url). Also, add a crawl_delay(url) method that returns the value of the Crawl-delay directive for a specified site.