Package: gain-requests-futures
Asynchronous Python HTTP Requests for Humans============================================.. image:: https://travis-ci.org/ross/requests-futures.png?branch=master:target: https://travis-ci.org/ross/requests-futuresSmall add-on for the python requests_ http library. Makes use of python 3.2's`concurrent.futures`_ or the backport_ for prior versions of python.The additional API and changes are minimal and strives to avoid surprises.The following synchronous code:.. code-block:: pythonfrom requests import Sessionsession = Session()# first requests starts and blocks until finishedresponse_one = session.get('http://httpbin.org/get')# second request starts once first is finishedresponse_two = session.get('http://httpbin.org/get?foo=bar')# both requests are completeprint('response one status: {0}'.format(response_one.status_code))print(response_one.content)print('response two status: {0}'.format(response_two.status_code))print(response_two.content)Can be translated to make use of futures, and thus be asynchronous by creatinga FuturesSession and catching the returned Future in place of Response. TheResponse can be retrieved by calling the result method on the Future:.. code-block:: pythonfrom requests_futures.sessions import FuturesSessionsession = FuturesSession()# first request is started in backgroundfuture_one = session.get('http://httpbin.org/get')# second requests is started immediatelyfuture_two = session.get('http://httpbin.org/get?foo=bar')# wait for the first request to complete, if it hasn't alreadyresponse_one = future_one.result()print('response one status: {0}'.format(response_one.status_code))print(response_one.content)# wait for the second request to complete, if it hasn't alreadyresponse_two = future_two.result()print('response two status: {0}'.format(response_two.status_code))print(response_two.content)By default a ThreadPoolExecutor is created with 2 workers. If you would like toadjust that value or share a executor across multiple sessions you can provideone to the FuturesSession constructor... code-block:: pythonfrom concurrent.futures import ThreadPoolExecutorfrom requests_futures.sessions import FuturesSessionsession = FuturesSession(executor=ThreadPoolExecutor(max_workers=10))# ...As a shortcut in case of just increasing workers number you can pass`max_workers` straight to the `FuturesSession` constructor:.. code-block:: pythonfrom requests_futures.sessions import FuturesSessionsession = FuturesSession(max_workers=10)FutureSession will use an existing session object if supplied:.. code-block:: pythonfrom requests import sessionfrom requests_futures.sessions import FuturesSessionmy_session = session()future_session = FuturesSession(session=my_session)That's it. The api of requests.Session is preserved without any modificationsbeyond returning a Future rather than Response. As with all futures exceptionsare shifted (thrown) to the future.result() call so try/except blocks should bemoved there.Canceling queued requests (a.k.a cleaning up after yourself)=========================If you know that you won't be needing any additional responses from futures thathaven't yet resolved, it's a good idea to cancel those requests. You can do thisby using the session as a context manager:.. code-block:: pythonfrom requests_futures.sessions import FuturesSessionwith FuturesSession(max_workers=1) as session:future = session.get('https://httpbin.org/get')future2 = session.get('https://httpbin.org/delay/10')future3 = session.get('https://httpbin.org/delay/10')response = future.result()In this example, the second or third request will be skipped, saving time andresources that would otherwise be wasted.Working in the Background=========================There is one additional parameter to the various request functions,background_callback, which allows you to work with the Response objects in thebackground thread. This can be useful for shifting work out of the foreground,for a simple example take json parsing... code-block:: pythonfrom pprint import pprintfrom requests_futures.sessions import FuturesSessionsession = FuturesSession()def bg_cb(sess, resp):# parse the json storing the result on the response objectresp.data = resp.json()future = session.get('http://httpbin.org/get', background_callback=bg_cb)# do some other stuff, send some more requests while this one worksresponse = future.result()print('response status {0}'.format(response.status_code))# data will have been attached to the response object in the backgroundpprint(response.data)Using ProcessPoolExecutor=========================Similarly to `ThreadPoolExecutor`, it is possible to use an instance of`ProcessPoolExecutor`. As the name suggest, the requests will be executedconcurrently in separate processes rather than threads... code-block:: pythonfrom concurrent.futures import ProcessPoolExecutorfrom requests_futures.sessions import FuturesSessionsession = FuturesSession(executor=ProcessPoolExecutor(max_workers=10))# ... use as before.. HINT::Using the `ProcessPoolExecutor` is useful, in cases where memoryusage per request is very high (large response) and cycling the interpretoris required to release memory back to OS.A base requirement of using `ProcessPoolExecutor` is that the `Session.request`,`FutureSession` and (the optional) `background_callback` all be pickle-able.This means that only Python 3.5 is fully supported, while Python versions3.4 and above REQUIRE an existing `requests.Session` instance to be passedwhen initializing `FutureSession`. Python 2.X and < 3.4 are currently notsupported... code-block:: python# Using python 3.4from concurrent.futures import ProcessPoolExecutorfrom requests import Sessionfrom requests_futures.sessions import FuturesSessionsession = FuturesSession(executor=ProcessPoolExecutor(max_workers=10),session=Session())# ... use as beforeIn case pickling fails, an exception is raised pointing to this documentation... code-block:: python# Using python 2.7from concurrent.futures import ProcessPoolExecutorfrom requests import Sessionfrom requests_futures.sessions import FuturesSessionsession = FuturesSession(executor=ProcessPoolExecutor(max_workers=10),session=Session())Traceback (most recent call last):...RuntimeError: Cannot pickle function. Refer to documentation: https://github.com/ross/requests-futures/#using-processpoolexecutor.. IMPORTANT::* Python >= 3.4 required* A session instance is required when using Python < 3.5* If sub-classing `FuturesSession` it must be importable (module global)* If using `background_callback` it too must be importable (module global)Installation============pip install requests-futures.. _`requests`: https://github.com/kennethreitz/requests.. _`concurrent.futures`: http://docs.python.org/dev/library/concurrent.futures.html.. _backport: https://pypi.python.org/pypi/futures