Supercharge Python’s Requests with Async IO & HTTPX | by Maulin Tolia | Towards Data Science

If you have some experience with Python, chances are good that you have used the Requests library at some point. It is a wonderful library for making HTTP requests and interacting with APIs.
The advantage of Requests is that it’s simple and syntactically sweet. The disadvantage is that it currently doesn’t work with Async IO — which can be really slow if you are dealing with many HTTP requests.
HTTPX is a new HTTP client with async support. It is very similar to Requests. In this tutorial, I will create a program with requests, give you an introduction to Async IO, and finally use Async IO & HTTPX to make the program much faster.
This tutorial assumes you have used Python’s Request library before.
In this example, I am going to use Google’s Books API to give me the names of books given their ISBN codes. Let's call it book finder.
Explanation
This is the flow of the book finder program:
I have a list of ISBN codes of various books stored in isbn_list. I use time.monotonic() to get the time I start making requests. Then I enumerate and iterate through the list.Next, I print the index and the name of the book returned by the print_book_name() function by passing the list index & ISBN code as a parameter. The reason I print the list index will be apparent in the async version.Finally, I calculate the difference between the current time and start time to get the time it took for the script to run.
I have a list of ISBN codes of various books stored in isbn_list. I use time.monotonic() to get the time I start making requests. Then I enumerate and iterate through the list.
Next, I print the index and the name of the book returned by the print_book_name() function by passing the list index & ISBN code as a parameter. The reason I print the list index will be apparent in the async version.
Finally, I calculate the difference between the current time and start time to get the time it took for the script to run.
This is the output of the script:
0 : The Hobbit1 : The Fellowship of the Ring2 : 19843 : Brave New World4 : The Player of Games5 : Watchmen6 : Crime and Punishment7 : Mistborn8 : The Adventures of Sherlock Holmes9 : The Great Gatsby10 : Moby-Dick11 : The Way of Kings12 : War and Peace13 : The Metamorphosis - Franz Kafka, New Edition14 : Liar's PokerTime Taken:6.764999999999418
It took 6.76 seconds to print the book names.
Async IO uses co-operative multitasking. Instead of creating threads for multitasking, you use a single event loop. Think of the event loop like a while loop that keeps switching between tasks in the program until there are no more tasks left.
The task switching takes place almost always when there is an I/O bound task(network calls, file I/O) in the program where the CPU remains idle while waiting for that resource to be available.
Picture this — You need to make an HTTP request and store the response data. With just 1 request it is simple. But what if you have to make 10s or 100s of requests in a loop like in the book finder program. When the program is synchronous the flow is:
Make the request.Wait for the response.Process the response.Repeat Steps 1–3 for all the requests.
Make the request.
Wait for the response.
Process the response.
Repeat Steps 1–3 for all the requests.
These steps are repeated for each request. The problem is that while the program is waiting for a response from the web server, the CPU is idle when it can be doing some other task.
With Async IO you can make the program switch to another task while it awaits a response from the web server. In our example, we will make all the HTTP requests concurrently and process each response as and when it comes. The program flow for 10 requests becomes like this:
Make all 10 requests sequentially & wait for the response.Process whichever response arrives first.
Make all 10 requests sequentially & wait for the response.
Process whichever response arrives first.
The advantage here is that when we can make all the 10 requests at once, some responses might come slowly while others may be a lot faster. We can process whichever response comes on a first come, first served basis which is much faster than in the synchronous version.
Concurrency is not parallelism. It's easy to get confused with the two terms.
Async IO was introduced in Python version 3.4 with the asyncio library. The async / await keywords have become part of Python's standard library since version 3.6. For this tutorial, I have used Python version 3.8. I will explain the implementation of asyncio by modifying the book finder program.
Note: The asyncio library changes quite a bit in every python version so please check your version before you get annoyed by syntax errors :)
Enough theory! Let's get back to writing some code.
First, install httpx with this command:
pip install httpx
Let’s modify our book finder program with asyncio & HTTPX. It may seem a little overwhelming at first but hang on. I will explain everything.
Explanation
An async function is called a coroutine. Coroutines are functions that are cooperative. It can only be executed inside an event loop. It can suspend its execution in the middle and pass control to the event loop which will then execute another coroutine. The execution is suspended when the coroutine needs to wait for a resource, in our case — the response from the server. It will resume execution when the response arrives.
Async programs are simply collections of coroutines. In python, the async keyword is added before the function definition to indicate that it's a coroutine.
I will go over the program flow:
I run asyncio.run() to start the event loop which will call the main() coroutine.Next, I gather a list of coroutine objects of get_book_name() with each ISBN code & index number as parameters in task_list. Notice that despite calling the coroutine, it did not execute. It instead returns a coroutine object. A coroutine object here, in simple words, is an instance of get_book_name() but needs to be scheduled inside the event loop to be executed. In order to schedule the coroutine object for execution you must use the await keyword before it. I will explain more about it in step 4.I use asyncio.gather() to execute the coroutine objects.asyncio.gather() is a coroutine that schedules awaitable objects(like coroutines) to be executed by the event loop. It also creates a list of objects returned from each coroutine call. In my coroutine(get_book_name()), I am not returning anything, so I don’t require this list.In get_book_name() I create an async client with HTTPX. Then I send the request to the same URL. Notice the await keyword before client.get(). This is because get() is also a coroutine. The await keyword lets you indicate where in the coroutine the execution needs to suspend. Here I use the await keyword before client.get() as I have to wait for the response from the server before proceeding.Note: Every coroutine is an awaitable object. Even asyncio.gather() is awaitable. This is because you can switch over to another task while it waits for a response. Every awaitable object requires the await keyword before it in order to execute it.After the client sends the request, the coroutine gives control of the execution back to the event loop which will call the next scheduledcoroutine object. The process will repeat for all the objects in task_list.As soon as a response arrives from any of the requests, the event loop will resume execution of that specific coroutine from where it was awaited. This means that the output will not be in the order of the requests as some responses will be faster than others. If a response arrives while a coroutine is being executed, then that responses’ coroutine gets scheduled waiting to resume execution.Once all the coroutines are executed, the event loop exits and the time taken is printed.
I run asyncio.run() to start the event loop which will call the main() coroutine.
Next, I gather a list of coroutine objects of get_book_name() with each ISBN code & index number as parameters in task_list. Notice that despite calling the coroutine, it did not execute. It instead returns a coroutine object. A coroutine object here, in simple words, is an instance of get_book_name() but needs to be scheduled inside the event loop to be executed. In order to schedule the coroutine object for execution you must use the await keyword before it. I will explain more about it in step 4.
I use asyncio.gather() to execute the coroutine objects.asyncio.gather() is a coroutine that schedules awaitable objects(like coroutines) to be executed by the event loop. It also creates a list of objects returned from each coroutine call. In my coroutine(get_book_name()), I am not returning anything, so I don’t require this list.
In get_book_name() I create an async client with HTTPX. Then I send the request to the same URL. Notice the await keyword before client.get(). This is because get() is also a coroutine. The await keyword lets you indicate where in the coroutine the execution needs to suspend. Here I use the await keyword before client.get() as I have to wait for the response from the server before proceeding.Note: Every coroutine is an awaitable object. Even asyncio.gather() is awaitable. This is because you can switch over to another task while it waits for a response. Every awaitable object requires the await keyword before it in order to execute it.
After the client sends the request, the coroutine gives control of the execution back to the event loop which will call the next scheduledcoroutine object. The process will repeat for all the objects in task_list.
As soon as a response arrives from any of the requests, the event loop will resume execution of that specific coroutine from where it was awaited. This means that the output will not be in the order of the requests as some responses will be faster than others. If a response arrives while a coroutine is being executed, then that responses’ coroutine gets scheduled waiting to resume execution.
Once all the coroutines are executed, the event loop exits and the time taken is printed.
When you run the program, you will get an output similar to this:
8: The Adventures of Sherlock Holmes0: The Hobbit14: Liar's Poker9: The Great Gatsby   11: The Way of Kings  4: The Player of Games5: Watchmen2: 198413: The Metamorphosis - Franz Kafka, New Edition10: Moby-Dick7: Mistborn1: The Fellowship of the Ring3: Brave New World6: Crime and PunishmentTime Taken:0.75
You can use the index to see the order of the response. It took just 0.75 seconds! That is a 900% increase. Of course, the results will vary based upon your internet speed but the async & HTTPX program in all cases will be much faster.
Before I conclude, it is important to set limits to the number of concurrent requests in HTTPX. A lot of APIs and websites have limits to the number of requests you can send per second or per minute. Luckily HTTPX has a function that can do this.
You can set limits by creating a Limits() object
limits = httpx.Limits(max_keepalive_connections=5, max_connections=10)
Then you need to pass the limit object to the httpx client.
client = httpx.Client(limits=limits)
HTTPX sets by default 10 keepalive connections and 100 max_connections at any given point. You can also set both to None, if you want your client to send as many requests as it can.
You can see how the asyncio library can speed up your programs. It can take a while for you to wrap your head around it but it is definitely worth the effort!
On the downside, it can make your program harder to read and complex to manage as you chain more coroutines.
Async IO has a lot of features to offer. You can also use it create TCP servers, speed up file I/O etc.