Add a mimimum TTL to the robots cache

Code to add a min_ttl variable to robots caches, default 60 seconds, to avoid pathological crawler behavior caused by honoring robots requests with a declared TTL of 0.