Package: cache_requests
cache_requestsSimple. Powerful. Persistent LRU caching for the requests library.FeaturesDocumentation:https://cache_requests.readthedocs.orgOpen Source:https://github.com/bionikspoon/cache_requestsPython version agnostic: tested against Python 2.7, 3.3, 3.4, 3.5 and PypyMIT licenseDrop in decorator for the requests library.Automatic timer based expiration on stored items (optional).Backed by yahoo’s powerfulredislite.Scalable with redis. Optionally accepts aredisconnection.Exposes the powerful underlyingMemoizedecorator to decorate any function.Tested with high coverage.Lightweight. Simple logic.Lightning fast.Jump start your development cycle.Collect and reuse entire response objects.InstallationAt the command line either via easy_install or pip$pipinstallcache_requests$easy_installcache_requestsOr, if you have virtualenvwrapper installed$mkvirtualenvcache_requests$pipinstallcache_requestsUninstall$pipuninstallcache_requestsUsageTo use cache_requests in a projectimportcache_requestsQuick StartTo usecache_requestsin a project>>>fromcache_requestsimportSession()requests=Session()# from python-requests.org>>>r=requests.get('https://api.github.com/user',auth=('user','pass'))>>>r.status_code200>>>r.headers['content-type']'application/json; charset=utf8'>>>r.encoding'utf-8'>>>r.textu'{"type":"User"...'>>>r.json(){u'private_gists':419,u'total_private_repos':77,...}Config OptionsDecorated Methodsmethod.exsets the default expiration (seconds) for new cache entries.method.rediscreates the connection to theredisorredislitedatabase. By default this is aredisliteconnection. However, a redis connection can be dropped in for easy scalability.cache_requests.Sessionexis shared between request methods.  They can be accessed bySession.cache.exorSession.get.ex, wheregetis therequests.getmethodBy default requests that return and error will not be cached.  This can be overridden by overriding theSession.cache.set_cache_cbto returnFalse.  The callback takes the response object as an argumentfromcache_requestsimportSessionrequests=Session()requests.cache.set_cache_db=lambda_:FalseBy default only autonomous methods are cached (get,head,options).  Each method can be setup to be cached using theSession.cacheconfig option.These methods are accessed through the Session objectsSession.cache.[methodname].
They can be overridden with theSession.cache.allsetting.For examplefromcache_requestsimportSessionrequests=Session()requests.cache.delete=True# cached, only called once.requests.delete('http://google.com')requests.delete('http://google.com')requests.cache.delete=True# not cached, called twice.requests.delete('http://google.com')requests.delete('http://google.com')# cache ALL methodsrequests.cache.all=True# don't cache any methodsrequests.cache.all=False# Use individual method cache options.requests.cache.all=NoneDefault settingsMethodCachedgetTrueheadTrueoptionsTruepostFalseputFalsepatchFalsedeleteFalseallNoneFunction Level ConfigCache BustingUse keywordbust_cache=Truein a memoized function to force reevaluation.Conditionally Set CacheUse keywordset_cacheto provide a callback.  The callback takes the results of function as an argument and must return abool. Alternatively,TrueandFalsecan be used.Use Case ScenariosDevelopment: 3rd Party APIsScenario:Working on a project that uses a 3rd party API or service.Things you want:A cache that persists between sessions and is lightning fast.Ability to rapidly explore the API and it’s parameters.Ability to inspect and debug response content.Ability to focus on progress.Perfect transition to a production environment.Things you don’t want:Dependency on network and server stability for development.Spamming the API.  Especially APIs with limits.Responses that change in non-meaningful ways.Burning energy with copypasta or fake data to run piece of your program.Slow. Responses.Make a request one time. Cache the results for the rest of your work session.importosifos.environ.get('ENV')=='DEVELOP':fromcache_requestsimportSessionrequest=Session(ex=60*60)# Set expiration, 60 minelse:importrequests# strange, complicated request you might makeheaders={"accept-encoding":"gzip, deflate, sdch","accept-language":"en-US,en;q=0.8"}payload=dict(sourceid="chrome-instant",ion="1",espv="2",ie="UTF-8",client="ubuntu",q="hash%20a%20dictionary%20python")response=requests.get('http://google.com/search',headers=headers,params=payload)# spam to prove a pointresponse=requests.get('http://google.com/search',headers=headers,params=payload)response=requests.get('http://google.com/search',headers=headers,params=payload)response=requests.get('http://google.com/search',headers=headers,params=payload)response=requests.get('http://google.com/search',headers=headers,params=payload)response=requests.get('http://google.com/search',headers=headers,params=payload)response=requests.get('http://google.com/search',headers=headers,params=payload)response=requests.get('http://google.com/search',headers=headers,params=payload)# tweak your query, we're exploring herepayload=dict(sourceid="chrome-instant",ion="1",espv="2",ie="UTF-8",client="ubuntu",q="hash%20a%20dictionary%20python2")# do you see what changed? the caching tool did.response=requests.get('http://google.com/search',headers=headers,params=payload)response=requests.get('http://google.com/search',headers=headers,params=payload)response=requests.get('http://google.com/search',headers=headers,params=payload)Production: Web ScrapingAutomatically expire old content.How often? After a day? A week? A Month? etc.  100% of this logic is built in with theSession.cache.exsetting.Effectively it can manage all of the time-based rotation.Perfect if you theres more data then what your API caps allow.One line of code to use aredisfull database.Tryredislite; it can handle quite a bit.  Theredisliteapi used by this module is 1:1 with the redis package.  Just replace the connection parameter/config value.redisis a drop in:connection=redis.StrictRedis(host='localhost',port=6379,db=0)requests=Session(connection=connection)*Everythingelsejustworks.There's no magic required.fromcache_requestsimportSessionconnection=redis.StrictRedis(host='localhost',port=6379,db=0)ex=7*24*60*60# 1 weekrequests=Session(ex=ex,connection=connection)foriinrange(1000)payload=dict(q=i)response=requests.get('http://google.com/search',params=payload)print(response.text)Usage: memoizefromcache_requestsimportMemoize@Memoize(ex=15*60)# 15 min, default, 60 mindefamazing_but_expensive_function(*args,**kwargs)print("You're going to like this")CreditsTools used in rendering this package:Cookiecutterbionikspoon/cookiecutter-pypackageforked fromaudreyr/cookiecutter-pypackageHistoryNext ReleaseStay tuned.4.0.0 (2015-12-25)Fix: Use MD5 for hash to avoid PYTHONHASHSEED issue.Fix: Give default dbfilename a more unique name, based on caller.BREAKING:MoveSession.exandSession.connectiontoSession.cacheconfig object.Updated examples.  New example demonstrates Memoize decorator.Updated requirements.3.0.0 (2015-12-22)Feature: Cache busting! Use keyword argumentbust_cache=Trueto force reevaluation.Feature: Session automatically skips caching error responses.Feature: Callback argument to decide if results should be cached.Feature: Decorated Session methods share a centralized configuration per session.BREAKING: Remove global config, in favor component level config.  Reasoning: Global config adds way too much complexity and adds too little value.  (Everything needs to lazy load the config at the last moment)Fix: Unique cache per function in shared db.Fix: Tweaks to keep the classes sub classable.Fix: Cleaned up tests.Updated requirements.2.0.0 (2015-12-12)API completely rewrittenNew API extendsrequestsinternals as opposed to monkeypatching.Entire package is redesigned to be more maintainable, more modular, and more usable.Dependencies are pinned.Tests are expanded.PY26 and PY32 support is dropped, because of dependency constraints.PY35 support is added.Docs are rewritten.Move towards idiomatic code.2.0.6 Fix broken coverage, broken rst render.1.0.0 (2015-04-23)First real release.Feature/ Unit test suite, very high coverage.Feature/redisliteintegration.Feature/ Documentation.https://cache-requests.readthedocs.org.Feature/ Exposed the beefed upMemoizedecorator.Feature/ Upgraded compatibility to:PY26PY27PY33PY34PYPYAdded examples and case studies.0.1.0 (2015-04-19)First release on PyPI.