tested robots crawl-delay directive
by forcing a 10.1 second delay for
diffbot.com in XmlDoc.cpp.
seemed to work after a few fixes.
however, it is ultimately only
an IP-based crawl delay, although
the delay applies to all subdomains
on the same domain, it's just that each IP
has its own timer for that delay.