Advanced Python: Python Programming guidelines For Multiprocessing
By: Farhad Malik
Tags: Technology, Fintech, Programming, Multiprocessing, Python

Advanced Python: Python Programming guidelines For Multiprocessing

Guidelines & Idioms When Using Multiple Processes In Python

One of the quickest and safest techniques to scale up the application is to launch multiple Python processes in an application. This also helps us by-pass the famous GIL issue.

Guidelines for multiple Python processes

The challenge of launching multiple processes is that it can cause unintended problems if the application is not designed appropriately.

I have written 12 top guidelines that I recommend everyone to follow. This article aims to outline the programming guidelines for multiple processing programming.

I recommend this article to everyone who is/or intends in using the Python programming language.

If you want to understand the Python programming language from the beginner to an advanced level then I highly recommend the article below:

Guideline 1

Regardless of the platform, whether you are using Windows or Linux, try to avoid sharing data between processes as much as possible. If you really have to share data then ensure the data size is as small as possible.

Guideline 2

If you have to share data then attempt to use queues or pipes to exchange data between the processes rather than using lower-level synchronization primitives.

Therefore, I don’t recommend using the lower level synchronization primitives that are available in the threading module.

Guideline 3

Make sure the arguments that you are passing to the processes are picklable. In particular, for Windows, ensure that the subclass Process instances are picklable when the start() method is called.

Guideline 4

Ensure your proxy objects are thread-safe.

If you are going to use a proxy object and your code has multiple threads then ensure the proxy object is protected with a lock. Having said that, if your application is multiprocessing in nature but uses a single thread within each process then it is not a problem and you don’t have to protect the proxy objects with a lock.

For more information on the multiprocessing library and to understand multiprocessing from the very basics, I highly recommend reading this article:

Guideline 5

Before your main process completes its execution, explicitly join all of the processes otherwise the processes can become zombie processes until a new process is started. This will ensure your application does not have any memory leaks.

Guideline 6

Don’t terminate a process abruptly. If your application uses shared resources that are being used by the terminated process then they will become broken and unavailable to be used by other processes.

Guideline 7

If your process is using a queue to exchange shared data between processes then make sure all of the items in the queue are removed before the process is joined.

If you want to read more about the synchronization primitives then read this article:

Guideline 8

Avoid creating global variables to share states. Global variables can be used as module-level constants but if the child processes attempt to access a global variable that is being updated by another variable then it can cause inconsistent behavior in the application.

I recommend passing the object as an argument to the constructor of the child process. This will prevent the garbage collector from removing the object as long as the child process is alive.

Guideline 9

Whenever we use the multiprocessing library to start a new process, it imports the main module in a new Python interpreter. Therefore it’s important to ensure the main module can be safely imported by a new Python interpreter without causing unintended side effects such as re-creating a new process and failing with a RuntimeError.

The key is to protect the entry point of the program by using:

if __name__ == '__main__:

# create new process

Guideline 10

As mentioned above, the shared objects that we pass to the queue/pipes are required to be picklable so that the child processes can use them. Having said that, I would encourage designing your solution in a way that you can inherit the shared object from the ancestor process. This will prevent deadlocks to occur in the application too.

Guideline 11

If your application creates a large number of tasks then use the Pool class to create the processes for you. The Pool class will take care of creating, allocating memory, and terminating processes for you. The Pool only allocates memory for the executing processes.

Guideline 12

Leave your multiprocessing application running and assess the memory profile. If the memory footprint is constantly increasing then it means the application is leaking memory. Profile your application by assessing the objects it is creating and storing memory in.

If you want to read more about how to profile your Python application then read this article:

Thank you for reading