Package: dlp-mpi
dlp_mpi - Data-level parallelism with mpi for pythonRun an serial algorithm on multiple examplesUse dlp_mpi to run the loop body in parallelUse dlp_mpi to run a function in parallel# python script.pyimporttimeexamples=list(range(10))results=[]forexampleinexamples:# Some heavy workload:# CPU or IOtime.sleep(0.2)result=example# Remember the resultsresults.append(result)# Summarize your experimentprint(sum(results))# mpiexec -np 8 python script.pyimporttimeimportdlp_mpiexamples=list(range(10))results=[]forexampleindlp_mpi.split_managed(examples):# Some heavy workload:# CPU or IOtime.sleep(0.2)result=example# Remember the resultsresults.append(result)results=dlp_mpi.gather(results)ifdlp_mpi.IS_MASTER:results=[resultforworker_resultsinresultsforresultinworker_results]# Summarize your experimentprint(results)# mpiexec -np 8 python script.pyimporttimeimportdlp_mpiexamples=list(range(10))results=[]defwork_load(example):# Some heavy workload:# CPU or IOtime.sleep(0.2)result=exampleforresultindlp_mpi.map_unordered(work_load,examples):# Remember the resultsresults.append(result)ifdlp_mpi.IS_MASTER:# Summarize your experimentprint(results)This package usesmpi4pyto provide utilities to parallelize algorithms that are applied to multiple examples.The core idea is: StartNprocesses and each process works on a subset of all examples.
To start the processesmpiexeccan be used. Most HPC systems support MPI to scatter the workload across multiple hosts. For the command, look in the documentation for your HPC system and search for MPI launches.Since each process should operate on different examples, MPI provides the variablesRANKandSIZE, whereSIZEis the number of workers andRANKis a unique identifier from0toSIZE - 1.
The easiest way to improve the execution time is to processexamples[RANK::SIZE]on each worker.
This is a round robin load balancing (dlp_mpi.split_round_robin).
A more advanced load balancing isdlp_mpi.split_managed, where one process manages the load and assigns a new task to a worker once he finishes the last task.When in the end of a program all results should be summarized or written in a single file, communication between all processes is nessesary.
For this purposedlp_mpi.gather(mpi4py.MPI.COMM_WORLD.gather) can be used. This function sends all data to the root process (Here,pickleis used for serialization).As an alternative to splitting the data, this package also provides amapstyle parallelization (see example in the beginning):
The functiondlp_mpi.map_unorderedcallswork_loadin parallel and executes theforbody in serial.
The communication between the processes is only theresultand the index to get theith example from the examples, i.e., the example aren't transferred between the processes.RuntimeWithout this package your code runs in serial.
The execution time of the following code snippets will be demonstrated by running it with this package.
Regarding the color: Theexamples = ...is the setup code.
Therefore, the code and the correspoding block representing the execution time it is blue in the code.This easiest way to parallelize the workload (dark orange) is to do a round robin assignment of the load:for example in dlp_mpi.split_round_robin(examples).
This function call is equivalent tofor example in examples[dlp_mpi.RANK::dlp_mpi.SIZE].
Thus, there is zero comunications between the workers.
Only when it is nessesary to do some final work on the results of all data (e.g. calculating average metrics) a communication is nessesary.
This is done with thegatherfunction.
This functions returns the worker results in a list on the master process and the worker process gets aNonereturn value.
Depending on the workload the round robin assingment can be suboptimal.
See the example block diagramm.
Worker 1 got tasks that are relative long.
So this worker used much more time than the others.To overcome the limitations of the round robin assignment, this package helps to use a manager to assign the work to the workers.
This optimizes the utilisation of the workers.
Once a worker finished an example, it requests a new one from the manager and gets one assigned.
Note: The communication is only which example should be processed (i.e. the index of the example) not the example itself.An alternative to splitting the iterator is to use amapfunction.
The function is then executed on a worker and the return value is sent back to the manager.
Be carefull, that the loop body is fast enough, otherwise it can be a bottleneck.
You should use the loop body only for book keeping, not for actual work load.
When a worker sends a task to the manager, the manager sends back a new task and enters the for loop body.
While the manager is in the loop body, he cannot react on requests of other workers, see the block diagramm:InstallationYou can install this package from pypi:pipinstalldlp_mpiTo check if the installation was successful, try the following command:$mpiexec-np4python-c'import dlp_mpi; print(dlp_mpi.RANK)'3012The command should print the numbers 0, 1, 2 and 3.
The order is random.
When that line prints 4 times a zero, something went wrong.This can happen, when you have nompiinstalled or the installation is brocken.
In a Debian-based Linux you can install it withsudo apt install libopenmpi-dev.
When you do not have the rights to install something withapt, you could also installmpi4pywithconda.
The abovepip installwill installmpi4pyfrompypi.
Be careful, that the installation fromcondamay conflict with your locally installedmpi.
Especially in High Performance Computing (HPC) environments this can cause troubles.