Package: requests-futures
Asynchronous Python HTTP Requests for HumansSmall add-on for the pythonrequestshttp library. Makes use of python 3.2’sconcurrent.futuresor thebackportfor prior versions of python.The additional API and changes are minimal and strives to avoid surprises.The following synchronous code:fromrequestsimportSessionsession=Session()# first requests starts and blocks until finishedresponse_one=session.get('http://httpbin.org/get')# second request starts once first is finishedresponse_two=session.get('http://httpbin.org/get?foo=bar')# both requests are completeprint('response one status:{0}'.format(response_one.status_code))print(response_one.content)print('response two status:{0}'.format(response_two.status_code))print(response_two.content)Can be translated to make use of futures, and thus be asynchronous by creating
a FuturesSession and catching the returned Future in place of Response. The
Response can be retrieved by calling the result method on the Future:fromrequests_futures.sessionsimportFuturesSessionsession=FuturesSession()# first request is started in backgroundfuture_one=session.get('http://httpbin.org/get')# second requests is started immediatelyfuture_two=session.get('http://httpbin.org/get?foo=bar')# wait for the first request to complete, if it hasn't alreadyresponse_one=future_one.result()print('response one status:{0}'.format(response_one.status_code))print(response_one.content)# wait for the second request to complete, if it hasn't alreadyresponse_two=future_two.result()print('response two status:{0}'.format(response_two.status_code))print(response_two.content)By default a ThreadPoolExecutor is created with 8 workers. If you would like to
adjust that value or share a executor across multiple sessions you can provide
one to the FuturesSession constructor.fromconcurrent.futuresimportThreadPoolExecutorfromrequests_futures.sessionsimportFuturesSessionsession=FuturesSession(executor=ThreadPoolExecutor(max_workers=10))# ...As a shortcut in case of just increasing workers number you can passmax_workersstraight to theFuturesSessionconstructor:fromrequests_futures.sessionsimportFuturesSessionsession=FuturesSession(max_workers=10)FutureSession will use an existing session object if supplied:fromrequestsimportsessionfromrequests_futures.sessionsimportFuturesSessionmy_session=session()future_session=FuturesSession(session=my_session)That’s it. The api of requests.Session is preserved without any modifications
beyond returning a Future rather than Response. As with all futures exceptions
are shifted (thrown) to the future.result() call so try/except blocks should be
moved there.Tying extra information to the request/responseThe most common piece of information needed is the URL of the request. This can
be accessed without any extra steps using therequestproperty of the
response object.fromconcurrent.futuresimportas_completedfrompprintimportpprintfromrequests_futures.sessionsimportFuturesSessionsession=FuturesSession()futures=[session.get(f'http://httpbin.org/get?{i}')foriinrange(3)]forfutureinas_completed(futures):resp=future.result()pprint({'url':resp.request.url,'content':resp.json(),})There are situations in which you may want to tie additional information to a
request/response. There are a number of ways to go about this, the simplest is
to attach additional information to the future object itself.fromconcurrent.futuresimportas_completedfrompprintimportpprintfromrequests_futures.sessionsimportFuturesSessionsession=FuturesSession()futures=[]foriinrange(3):future=session.get('http://httpbin.org/get')future.i=ifutures.append(future)forfutureinas_completed(futures):resp=future.result()pprint({'i':future.i,'content':resp.json(),})Canceling queued requests (a.k.a cleaning up after yourself)If you know that you won’t be needing any additional responses from futures that
haven’t yet resolved, it’s a good idea to cancel those requests. You can do this
by using the session as a context manager:fromrequests_futures.sessionsimportFuturesSessionwithFuturesSession(max_workers=1)assession:future=session.get('https://httpbin.org/get')future2=session.get('https://httpbin.org/delay/10')future3=session.get('https://httpbin.org/delay/10')response=future.result()In this example, the second or third request will be skipped, saving time and
resources that would otherwise be wasted.Iterating over a list of requests responsesWithout preserving the requests order:fromconcurrent.futuresimportas_completedfromrequests_futures.sessionsimportFuturesSessionwithFuturesSession()assession:futures=[session.get('https://httpbin.org/delay/{}'.format(i%3))foriinrange(10)]forfutureinas_completed(futures):resp=future.result()print(resp.json()['url'])Working in the BackgroundAdditional processing can be done in the background using requests’shooksfunctionality. This can be useful for shifting work out of the foreground, for
a simple example take json parsing.frompprintimportpprintfromrequests_futures.sessionsimportFuturesSessionsession=FuturesSession()defresponse_hook(resp,*args,**kwargs):# parse the json storing the result on the response objectresp.data=resp.json()future=session.get('http://httpbin.org/get',hooks={'response':response_hook,})# do some other stuff, send some more requests while this one worksresponse=future.result()print('response status{0}'.format(response.status_code))# data will have been attached to the response object in the backgroundpprint(response.data)Hooks can also be applied to the session.frompprintimportpprintfromrequests_futures.sessionsimportFuturesSessiondefresponse_hook(resp,*args,**kwargs):# parse the json storing the result on the response objectresp.data=resp.json()session=FuturesSession()session.hooks['response']=response_hookfuture=session.get('http://httpbin.org/get')# do some other stuff, send some more requests while this one worksresponse=future.result()print('response status{0}'.format(response.status_code))# data will have been attached to the response object in the backgroundpprint(response.data)pprint(response.data)A more advanced example that adds anelapsedproperty to all requests.frompprintimportpprintfromrequests_futures.sessionsimportFuturesSessionfromtimeimporttimeclassElapsedFuturesSession(FuturesSession):defrequest(self,method,url,hooks=None,*args,**kwargs):start=time()ifhooksisNone:hooks={}deftiming(r,*args,**kwargs):r.elapsed=time()-starttry:ifisinstance(hooks['response'],(list,tuple)):# needs to be first so we don't time other hooks executionhooks['response'].insert(0,timing)else:hooks['response']=[timing,hooks['response']]exceptKeyError:hooks['response']=timingreturnsuper(ElapsedFuturesSession,self)\.request(method,url,hooks=hooks,*args,**kwargs)session=ElapsedFuturesSession()future=session.get('http://httpbin.org/get')# do some other stuff, send some more requests while this one worksresponse=future.result()print('response status{0}'.format(response.status_code))print('response elapsed{0}'.format(response.elapsed))Using ProcessPoolExecutorSimilarly toThreadPoolExecutor, it is possible to use an instance ofProcessPoolExecutor. As the name suggest, the requests will be executed
concurrently in separate processes rather than threads.fromconcurrent.futuresimportProcessPoolExecutorfromrequests_futures.sessionsimportFuturesSessionsession=FuturesSession(executor=ProcessPoolExecutor(max_workers=10))# ... use as beforeHintUsing theProcessPoolExecutoris useful, in cases where memory
usage per request is very high (large response) and cycling the interpreter
is required to release memory back to OS.A base requirement of usingProcessPoolExecutoris that theSession.request,FutureSessionall be pickle-able.This means that only Python 3.5 is fully supported, while Python versions
3.4 and above REQUIRE an existingrequests.Sessioninstance to be passed
when initializingFutureSession. Python 2.X and < 3.4 are currently not
supported.# Using python 3.4fromconcurrent.futuresimportProcessPoolExecutorfromrequestsimportSessionfromrequests_futures.sessionsimportFuturesSessionsession=FuturesSession(executor=ProcessPoolExecutor(max_workers=10),session=Session())# ... use as beforeIn case pickling fails, an exception is raised pointing to this documentation.# Using python 2.7fromconcurrent.futuresimportProcessPoolExecutorfromrequestsimportSessionfromrequests_futures.sessionsimportFuturesSessionsession=FuturesSession(executor=ProcessPoolExecutor(max_workers=10),session=Session())Traceback(mostrecentcalllast):...RuntimeError:Cannotpicklefunction.Refertodocumentation:https://github.com/ross/requests-futures/#using-processpoolexecutorImportantPython >= 3.4 requiredA session instance is required when using Python < 3.5If sub-classingFuturesSessionit must be importable (module global)Installationpip install requests-futures