Support Crawl-Delay directive
Respect crawl-delay directive of the * record if it's not set directly on the record for the UA