reclosedev/requests-cache — pull request #4 by mapio (2012-05-10 09:02:07+00:00)
Title: Added a wait option

I am using this nice tool to do some web scraping of sites that are quite sensitive to politeness and access rates.

I added a wait option that works like this: if the request in in cache, it is immediately returned. On the other hand, if the request is actually performed on the net (and ends in a cache-able response), the configured time is waited before returning to the caller.

In this way a very simple "throttling" is transparently available to the library user.

Much better solution can be implemented, for instance keeping time of the last request time per contacted host, or using multiprocessing and futures…

My goal with the proposed solution is to favor simplicity and reliability.

Hope this helps,
Massimo