seomoz/reppy — pull request #19 by DavidBarts (2014-12-09 22:47:09+00:00)
Title: OK, take 2, updated master.

Simple stuff: provide an honor_ttl option to fetch. Default is true, the legacy behavior. But there's now a way to request the cache always be used even if the data therein is stale.

Reason is I need to sometimes return "stale" information in a multithreaded crawler I'm writing -- in the case where multiple threads are waiting on the same robots.txt file, all waiting threads should get it, even if the server marks it pre-expired (as some servers do).