Add non-blocking options to allowed? and disallowed? methods and added a method to return the value of the crawl delay directive for a given URL.