seomoz/reppy — pull request #5 by DavidBarts (2013-08-26 23:41:29+00:00)
Title: Add a mimimum TTL to the robots cache

Code to add a min_ttl variable to robots caches, default 60 seconds, to avoid pathological crawler behavior caused by honoring robots requests with a declared TTL of 0.