Improved handling of robots.txt Crawl-delay directive.