Beginning Python Programming — Part 14
By: Bob Roebling
Tags: Python3, Asynchronous, Programming, Python, Threading

ThreadPoolExecutor

Sometimes you need to customize how many threads should be used for a task, such as limiting the number of concurrent threads that should be used over a series of tasks.

To illustrate this, look at the following example, which is just a refactor of the example above:

Here we end up creating four different threads to process the same data. I just decided to use loops to manage the complexity of re-writing the same code over and over again (DRY principle).

Let’s say we have multiple tasks that need to run on a background thread. Each of these tasks requires some preliminary information that must be completed before the respective task begins. A basic web crawler would be a great example.

On small websites, we might be able to get by with threading but on large sites, we definitely need to limit the number of threads we are using so we don’t run into performance issues. You can only go so fast.

Here is a refactor using ThreadPoolExecutor based on an example found here:

We have a few more imports at the top. We include threading because we need to get the active_count() of threads when our program starts and we need to be able to figure out which thread we are currently using for our task. We include time because we want to calculate the run time of the application when we adjust how many threads we are using. Last, we use from concurrent.futures import ThreadPoolExecutor this allows us to import the ThreadPoolExecutor exclusively, instead of the entire concurrent library. This is what we will call to set up an executor later to run all of our tasks.

I start off creating a list of numbers that will contain values between 100,000 and 1,000,000 in increments of 10,000. We will use this to pass consistent values to each task between runs. I initially started off using randint from the random library, but realized that one run might get nothing but low values while another might get nothing but high values. This would skew our results.

We then create our task . This is similar to the count function above except we pass in the number we want to use as our max number. We let everyone know we started a task on a new thread and provide the name. Since the name is ThreadPoolExecutor-x-y , where x is the current thread pool and y is the current thread, I only use the last three values of the string.

We initialize result to 0 and start our loop to sum all of the numbers. We print the result before printing that the current_thread() has finished.

Our main function comes next. In it, we first create an executor object from ThreadPoolExecutor passing in the maximum number of workers we wish to use. We can also do this by using the named parameter max_workers=10 . If you leave this blank, you end up with the number of processors on the machine multiplied by five. On my computer, I have 8 logical processors * 5 threads = 40 threads.

Normally, we’d call executor.submit(task) for each task, but in this scenario, we are passing a variable to the function, in this case, we are using executor.map(task, numbers) to do all of this work for us. What happens is the Interpreter first checks to get the length of numbers then for each number in numbers it passes the value to the number parameter of a task . Once it has finished, it will then submit the task using the executor .

We then have our if __name__ == "__main__” statement, which checks to see if this file was invoked. When I initially created this program, I didn’t include the next line, which gets the current number of active threads, and my program never finished. When I printed the number of active threads, I found that I was already using three threads for this program.

Because of this, I needed to get the active number of threads so I could keep the program from running the time elapsed line at the bottom immediately. Think of this as a basic semaphore, which is used for ensuring synchronization of multiple threads. Python has these built-in, but the example is more comfortable to digest this way.

Moving down, I get the current time so we have something to calculate the elapsed time against later. Then I call main() which kicks off the business logic of our application. Once all of the work is done, we determine the elapsed time in seconds and print it out to the screen.

After some testing with various numbers, I did some data science on the numbers to show a correlation between the number of threads used and the amount of time it took to run.