Supercharge Python’s Requests with Async IO & HTTPX
By: Maulin Tolia
Tags: Http Request, API, Python, Web Scraping, Asyncio

Quick Introduction to Async IO

Async IO uses co-operative multitasking. Instead of creating threads for multitasking, you use a single event loop. Think of the event loop like a while loop that keeps switching between tasks in the program until there are no more tasks left.

The task switching takes place almost always when there is an I/O bound task(network calls, file I/O) in the program where the CPU remains idle while waiting for that resource to be available.

Picture this — You need to make an HTTP request and store the response data. With just 1 request it is simple. But what if you have to make 10s or 100s of requests in a loop like in the book finder program. When the program is synchronous the flow is:

Synchronous Flow — Created by the author

Make the request. Wait for the response. Process the response. Repeat Steps 1–3 for all the requests.

These steps are repeated for each request. The problem is that while the program is waiting for a response from the web server, the CPU is idle when it can be doing some other task.

With Async IO you can make the program switch to another task while it awaits a response from the web server. In our example, we will make all the HTTP requests concurrently and process each response as and when it comes. The program flow for 10 requests becomes like this:

Async Flow (The responses will not be in order) — Created by the author

Make all 10 requests sequentially & wait for the response. Process whichever response arrives first.

The advantage here is that when we can make all the 10 requests at once, some responses might come slowly while others may be a lot faster. We can process whichever response comes on a first come, first served basis which is much faster than in the synchronous version.

Concurrency is not parallelism. It's easy to get confused with the two terms.

Async IO was introduced in Python version 3.4 with the asyncio library. The async / await keywords have become part of Python's standard library since version 3.6. For this tutorial, I have used Python version 3.8. I will explain the implementation of asyncio by modifying the book finder program.

Note: The asyncio library changes quite a bit in every python version so please check your version before you get annoyed by syntax errors :)

Enough theory! Let's get back to writing some code.

HTTPX & Async IO

First, install httpx with this command:

pip install httpx

Let’s modify our book finder program with asyncio & HTTPX. It may seem a little overwhelming at first but hang on. I will explain everything.

Explanation

An async function is called a coroutine. Coroutines are functions that are cooperative. It can only be executed inside an event loop. It can suspend its execution in the middle and pass control to the event loop which will then execute another coroutine. The execution is suspended when the coroutine needs to wait for a resource, in our case — the response from the server. It will resume execution when the response arrives.

Async programs are simply collections of coroutines. In python, the async keyword is added before the function definition to indicate that it's a coroutine.

I will go over the program flow:

I run asyncio.run() to start the event loop which will call the main() coroutine. Next, I gather a list of coroutine objects of get_book_name() with each ISBN code & index number as parameters in task_list . Notice that despite calling the coroutine, it did not execute. It instead returns a coroutine object. A coroutine object here, in simple words, is an instance of get_book_name() but needs to be scheduled inside the event loop to be executed. In order to schedule the coroutine object for execution you must use the await keyword before it. I will explain more about it in step 4. I use asyncio.gather() to execute the coroutine objects. asyncio.gather() is a coroutine that schedules awaitable objects(like coroutines) to be executed by the event loop. It also creates a list of objects returned from each coroutine call. In my coroutine( get_book_name() ), I am not returning anything, so I don’t require this list. In get_book_name() I create an async client with HTTPX. Then I send the request to the same URL. Notice the await keyword before client.get() . This is because get() is also a coroutine. The await keyword lets you indicate where in the coroutine the execution needs to suspend. Here I use the await keyword before client.get() as I have to wait for the response from the server before proceeding.

Note: Every coroutine is an awaitable object. Even asyncio.gather() is awaitable. This is because you can switch over to another task while it waits for a response. Every awaitable object requires the await keyword before it in order to execute it. After the client sends the request, the coroutine gives control of the execution back to the event loop which will call the next scheduled coroutine object. The process will repeat for all the objects in task_list . As soon as a response arrives from any of the requests, the event loop will resume execution of that specific coroutine from where it was awaited. This means that the output will not be in the order of the requests as some responses will be faster than others. If a response arrives while a coroutine is being executed, then that responses’ coroutine gets scheduled waiting to resume execution. Once all the coroutines are executed, the event loop exits and the time taken is printed.

When you run the program, you will get an output similar to this:

8: The Adventures of Sherlock Holmes

0: The Hobbit

14: Liar's Poker

9: The Great Gatsby

11: The Way of Kings

4: The Player of Games

5: Watchmen

2: 1984

13: The Metamorphosis - Franz Kafka, New Edition

10: Moby-Dick

7: Mistborn

1: The Fellowship of the Ring

3: Brave New World

6: Crime and Punishment

Time Taken:0.75

You can use the index to see the order of the response. It took just 0.75 seconds! That is a 900% increase. Of course, the results will vary based upon your internet speed but the async & HTTPX program in all cases will be much faster.

Setting Limits to the HTTPX Client

Before I conclude, it is important to set limits to the number of concurrent requests in HTTPX. A lot of APIs and websites have limits to the number of requests you can send per second or per minute. Luckily HTTPX has a function that can do this.

You can set limits by creating a Limits() object

limits = httpx.Limits(max_keepalive_connections=5, max_connections=10)

Then you need to pass the limit object to the httpx client.

client = httpx.Client(limits=limits)

HTTPX sets by default 10 keepalive connections and 100 max_connections at any given point. You can also set both to None , if you want your client to send as many requests as it can.

Conclusion

You can see how the asyncio library can speed up your programs. It can take a while for you to wrap your head around it but it is definitely worth the effort!

On the downside, it can make your program harder to read and complex to manage as you chain more coroutines.

Async IO has a lot of features to offer. You can also use it create TCP servers, speed up file I/O etc.

Useful Links