Beginning Python Programming — Part 15
By: Bob Roebling
Tags: Async, Python3, Programming, Python, Multiprocessing

In the previous article, we covered multi-threading.

Here we will be covering another library provided by Python called multiprocessing . Like threading , multiprocessing allows us to run code concurrently; however, this code runs on multiple processors. Let’s start with a definition.

Global Interpreter Lock (GIL) — used to ensure that Python bytecode only runs on one thread at a time when locked. If released, it allows using multiple cores for processing.

So far, we’ve been able to dodge this complexity, as we have only been dealing with single core computations. The GIL is vital to ensure the safety of concurrent access; in layman’s terms, multiple tasks do not cause problems when running at the same time.

In our previous examples, we ended up having two lines printed to the console at the same time. The GIL could be used to clean this up by ensuring only one process can write to the console at a time.

While the GIL works to protect us, it limits us from the parallel processing that we get by using multiple cores, which in turn causes performance issues. It doesn’t mean “never use the multiprocessing,” it just means that we should only use multi-processing when we know exactly how we want our code to perform.

Multiprocessing

multiprocessing is best used when we are performing CPU-based calculations. CPU calculations might be Bitcoin mining (the slow way) or creating rainbow tables for some extreme examples. Some more practical examples are photo and video editing, music composition, or creating hash tables (data mining stuff).

Here’s an example you’ll recognize from the previous post slightly changed to work for multiprocessing.

Instead of importing threading and concurrent.futures , we only need to import multiprocessing . The next thing that has changed is that I used multiprocessing.current_process().pid instead of threading.active_count().name[-3:] . This gets the Process ID of the process that spun from our main process and reports it back to the console, so we know where to find our process if we ever need to terminate it later. Everything else in our task function is the same.

In main , we changed our executor, which once stood as an instance of ThreadPoolExecutor , but we changed it to multiprocessing.Pool() . Here, we specified only a single process should be used. If I left it blank, it would return the total number of CPUs available in my machine using os.cpu_count() .

Like threading, I am using map to map each number to its own task . If I have more tasks than processes, they will queue up and wait for the previous task to complete. Finally, I added executor.close() because I had a runaway process. close() cleans up the pool and ensures it is properly stopped.

In if __name__ == “__main__" , we removed the line current_threads = threading.active_count() . We did not replace it with the equivalent line for multiprocessing, which would be current_children = multiprocessing.active_children() , because our executor will block the main thread until it is completed. We could run this using a more true apply_async call (similar to start ), but for our purposes, we won’t need it here.

I also continue to use time to check our execution run time as we did previously.

Here are the results of testing multiple processes: