Package: txrequests
Asynchronous Python HTTP Requests for HumansSmall add-on for the pythonrequestshttp library. Makes use twisted’s ThreadPool,
so that the requests’API returns deferredThe additional API and changes are minimal and strives to avoid surprises.The following synchronous code:fromrequestsimportSessionsession=Session()# first requests starts and blocks until finishedresponse_one=session.get('http://httpbin.org/get')# second request starts once first is finishedresponse_two=session.get('http://httpbin.org/get?foo=bar')# both requests are completeprint('response one status:{0}'.format(response_one.status_code))print(response_one.content)print('response two status:{0}'.format(response_two.status_code))print(response_two.content)Can be translated to make use of futures, and thus be asynchronous by creating
a FuturesSession and catching the returned Future in place of Response. The
Response can be retrieved by calling the result method on the Future:fromtxrequestsimportSessionfromtwisted.internetimportdefer@defer.inlineCallbacksdefmain():# use with statement to cleanup session's threadpool, and connectionpool after use# you can also use session.close() if want to use session for long term usewithSession()assession:# first request is started in backgroundd1=session.get('http://httpbin.org/get')# second requests is started immediatelyd2=session.get('http://httpbin.org/get?foo=bar')# wait for the first request to complete, if it hasn't alreadyresponse_one=yieldd1print('response one status:{0}'.format(response_one.status_code))print(response_one.content)# wait for the second request to complete, if it hasn't alreadyresponse_two=yieldd2print('response two status:{0}'.format(response_two.status_code))print(response_two.content)By default a ThreadPool is created with 4 max workers. If you would like to
adjust that value or share a threadpool across multiple sessions you can provide
one to the Session constructor.fromtwisted.python.threadpoolimportThreadPoolfromtxrequestsimportSessionsession=FuturesSession(pool=ThreadPool(maxthreads=10))# ...As a shortcut in case of just increasing workers number you can passminthreadsand/ormaxthreadsstraight to theSessionconstructor:fromtxrequestsimportSessionsession=Session(maxthreads=10)That’s it. The api of requests.Session is preserved without any modifications
beyond returning a Deferred rather than Response. As with all futures exceptions
are shifted to the deferred errback.Working in the BackgroundThere is one additional parameter to the various request functions,
background_callback, which allows you to work with the Response objects in the
background thread. This can be useful for shifting work out of the foreground,
for a simple example take json parsing.frompprintimportpprintfromtxrequestsimportSessionfromtwisted.internetimportdefer@defer.inlineCallbacksdefmain():withSession()assession:defbg_cb(sess,resp):# parse the json storing the result on the response objectresp.data=resp.json()returnrespd=session.get('http://httpbin.org/get',background_callback=bg_cb)# do some other stuff, send some more requests while this one worksresponse=yielddprint('response status{0}'.format(response.status_code))# data will have been attached to the response object in the backgroundpprint(response.data)Installationpip install txrequestsCreditstxrequests is based onrequests_future, from Ross McFarland