Disallow robot from crawling. They puch up the queue like crazy.
Second level of disallowing robots.
Added some file/access stats to the cache. This is th first step in evicting cache items.