Speed Up Your Algorithms Part 3 — Parallel-ization | by Puneet Grover | Towards Data Science

This is the third post in a series I am writing. All posts are here:
Speed Up your Algorithms Part 1 — PyTorchSpeed Up your Algorithms Part 2 — NumbaSpeed Up your Algorithms Part 3 — ParallelizationSpeed Up your Algorithms Part 4 — Dask
Speed Up your Algorithms Part 1 — PyTorch
Speed Up your Algorithms Part 2 — Numba
Speed Up your Algorithms Part 3 — Parallelization
Speed Up your Algorithms Part 4 — Dask
And these goes with Jupyter Notebooks available here:
[Github-SpeedUpYourAlgorithms] and [Kaggle]
IntroductionPool and ProcessThreadingDasktorch.multiprocessingFurther ReadingReferences
Introduction
Pool and Process
Threading
Dask
torch.multiprocessing
Further Reading
References
NOTE:This post goes with Jupyter Notebook available in my Repo on Github:[SpeedUpYourAlgorithms-Parallelization]and on Kaggle:[SpeedUpYourAlgorithms-Parallelization]
With Data increasing exponentially with time and growth in processor’s computing power coming to a hold, we need to find ways to process data efficiently and effectively. What should we do?
GPU is one solution, and it is very effective. But, GPU’s were not made for Machine Learning purposes, they were specifically made for complex image processing and gaming. We made our algorithms to work on existing GPUs and it actually paid off. Now, Google has introduced a new device, called TPU (Tensor Processing Unit), which is tailored for machine learning workloads on TensorFlow and the results do look promising. And Nvidia is also not backing off.1
But we will hit the ceiling at some point in the future. Even if we take any of the enormous datasets available today, a single machine or computing unit will not be sufficient to handle such load. We will have to use multiple machines to do our task(s). We will have to Parallelize our task(s).
In this post we will look into some of the methods which you will work with most of the time in Python. And then a little introduction to Dask and torch.multiprocessing.
Both Pool and Process methods of multiprocessing library of Python initiates a new process for our task, but in a different way. Process makes only one process per call:
import multiprocessing as mpp = mp.Process(target= ##target-function,               args=   ##args-to-func)# This call will make only one process, which will process# target-function with given arguments in background.
But the process still hasn’t started. To start it you have to do:
p.start()
Now you can either leave it here, or check if the process is complete or not by:
p.join()# Now it will wait for process to complete.
There can be many uses of not checking if the process has completed or not. For example, in a Client-Server Application where the probability of packet being lost, or a non-responding process is really low, we can neglect it which can give us considerable speedup. [Depends on Application’s process]
For multiple processes you will have to make multiple Processs. You can make as many you like. All of them will start when you invoke .start() on them.
processes =[mp.Process(target=func, args=(a, b)) for (a, b) in list]for p in processes: p.start()for p in processes: p.join()
On the other hand a Pool initiates a fixed number of processes and then we can assign some tasks to those processes. So, at a particular instance of time only a fixed amount of processes will be running and the rest will be waiting. The number of processes are mostly chosen as number of cores of your device, which is also the default behavior if you leave the argument empty.
pool = mp.Pool(processes=2)
Now there are many methods which you can apply to use this Pool. Ones we can get away with, in Data Science, are Pool.apply and Pool.map, because they return the result as soon as task(s) are complete. Pool.apply takes only one argument and uses only one process, whereas Pool.map takes many arguments and puts them onto our Pool of processes.
results = [pool.apply(func, (x)) for x in X]# Or results = pool.map(func, (arg)) # Takes only one argument
But Pool.map takes only one argument (iterable) which it divides into number of chunks. To send in many arguments you can do something like this.
Considering our previous example of Client-Server Application, here max number of processes to be running are predefined, so if we have a lot of requests/packets, only n (max processes in Pool) of them will be running at a time, and others will be in queue for one of the processes slot waiting for its turn.
# How can we use it with Data Frame?# A: You can use some parallelizable functiondf.shape# (100, 100)dfs = [df.iloc[i*25:i*25+25, 0] for i in range(4)]with Pool(4) as p:    res = p.map(np.exp, dfs)for i in range(4): df.iloc[i*25:i*25+25, 0] = res[i]# It can come in handy for preprocessing of data.
What to use and when?2
If you have a lot of tasks and not many of them are compute intensive, you should use a Process. Because if they are compute intensive, they might clog your CPU and your system might crash. And they won’t have to wait in queue for their chance if your system can handle them all at once.
And when you have fixed amount of tasks and they are very compute intensive, you should use a Pool. Because if you set them loose all at once, your system might crash.
Threading! In python?
Threading in python has a bad reputation. And people are correct. Threading actually doesn’t work as it should most of the time. So what is the problem?
The problem is GIL (Global Interpreter Lock). GIL was introduced early on Python’s development when there wasn’t even concept of Threads in Operating Systems. It was chosen for its simplicity.
GIL allows only one CPU bound process at a time. That is, it gives access of python interpreter to only one thread at a time. So, a threadLocks the whole interpreter until its complete.3
For single-threaded programs it was fast as there was only one Lock to maintain. And as python became popular it became hard to effectively put out GIL and not harm all the dependent applications. And that’s why its still there.
But, you can still use multi-threading parallel(y) if your task is not CPU bound. That is you can use multiple threads and get speedup if your task is I/O bound. Because most of the time these tasks are waiting for response from other agent (like disk etc.) and in that time they can release the lock, leaving it for other tasks to acquire it in meantime.4
NOTE: (From official page here)The GIL is controversial because it prevents multithreaded CPython programs from taking full advantage of multiprocessor systems in certain situations. Note that potentially blocking or long-running operations, such as I/O, image processing, and NumPy number crunching, happen outside the GIL. Therefore it is only in multithreaded programs that spend a lot of time inside the GIL, interpreting CPython bytecode, that the GIL becomes a bottleneck.
So, if your task is IO bound, something like downloading some data from server, read/write to disk etc., you can use multiple threads and get speedup.
from threading import Thread as timport queueq = queue.Queue()  # For putting and getting results of threadfunc_ = lambda q, args: q.put(func(args))threads = [t(target=func_, args=(q, args)) for args in args_array]for t in threads: t.start()for t in threads: t.join()res = []for t in threads: res.append(q.get()) # These results won't necessarily be in order
To save results from Threads, you can use something like Queue. And for that you will have to define your function as shown above or you can use Queue.put() inside your function, but for that you will have to change your function definition to include Queue as argument.6
Now, your results in queue won’t necessarily be in order. If you want your results to be in order you can pass in some counter as arguments, as ids, and then use those ids to identify where the result came from.
threads = [t(func_, args = (i, q, args)) for i, args in                                          enumerate(args_array)]# And update function accordinglyNOTE:Multiprocessing with Pandas 'read.csv' method doesn't give much speedup for some reason. As an alternative you can use Dask.
Threads vs Processes?7
A process is heavy-weight as it may contain many threads of its own (contains atleast one) and it has its own memory space allocated, whereas threads are light-weight as it works on memory area of parent process and thus faster to make.
Communication between threads inside a process is easier because they share same memory space., whereas communication between processes (IPC-Inter Process Communication) is slower. But then again, threads sharing same data can get into race condition, and should be taken care of using Locks or similar solutions.
Dask is a parallel computing library which doesn’t just help parallelize existing Machine Learning tools (Pandas andNumpy)[i.e. using High Level Collection], but also helps parallelize low level tasks/functions and can handle complex interactions between these functions by making a tasks’ graph.[i.e. using Low Level Schedulers] This is similar to Threading or multiprocessing modules of Python.
They also have a separate Machine Learning library, dask-ml, which has integration with existing libraries such as sklearn, xgboost and tensorflow.
from dask import delayed as delay@delaydef add(x, y):    return x+y@delaydef sq(x):    return x**2# Now you can use these functions any way you want, Dask will # parallelize your execution. And as the name suggest Dask # will not execute your function callings right away, rather# it will make a computational graph depending on the way you are# calling functions on inputs and intermediate results. To compute# final result:result.compute()
Dask has a inherently parallel nature of doing everything. For how it handles DataFrames, you can think of it as a Divide and Conquer approach, where it divides your DataFrame into chunks and then apply your given function in parallel.
df = dask.DataFrame.read_csv("BigFile.csv", chunks=50000)# Your DataFrame has been divided into chunks and every function# you apply will be applied to all chunks separately and in # parallel.# It has most of Pandas functions which you can use:agg = df.groupby(["column"]).aggregate(["sum", "mean"])agg.columns = new_column_namesdf_new = df.merge(agg.reset_index(), on="column", how="left")# It have not compute result up until now,# but with .compute() it will compute now in parallel.df_new.compute().head()
They also have interface for running them on cluster of machines.
For full introduction on Dask, see my post here.
torch.multiprocessing is a wrapper around Python multiprocessing module and its API is 100% compatible with original module. So you can use Queue's, Pipe's, Array's etc. which are in Python’s multiprocessing module here. To add to that, to make it faster they have added a method, share_memory_(), which allows data to go into a state where any process can use it directly and so passing that data to as argument to different processes won’t make copy of that data.
You can share Tensors, model’s parameters, and you can share them on CPU or GPU as you like.
Warning from Pytorch: (Regarding sharing on GPU)  CUDA API requires that the allocation exported to other processes remains valid as long as it’s used by them. You should be careful and ensure that CUDA tensors you shared don’t go out of scope as long as it’s necessary. This shouldn’t be a problem for sharing model parameters, but passing other kinds of data should be done with care. Note that this restriction doesn’t apply to shared CPU memory.
You can use methods above in “Pool and Process” section here, and to get more speedup you can use share_memory_() method to share a Tensor(say) among all processes without being copied.
# Training a model using multiple processes:import torch.multiprocessing as mpdef train(model):    for data, labels in data_loader:        optimizer.zero_grad()        loss_fn(model(data), labels).backward()        optimizer.step()  # This will update the shared parametersmodel = nn.Sequential(nn.Linear(n_in, n_h1),                      nn.ReLU(),                      nn.Linear(n_h1, n_out))model.share_memory() # Required for 'fork' method to workprocesses = []for i in range(4): # No. of processes    p = mp.Process(target=train, args=(model,))    p.start()    processes.append(p)for p in processes: p.join()
You can also work with a cluster of machines. For more info see here.
NOTE:For little introduction (kind of) on usage of Pycuda, see Jupyter Notebook's PyCuda section here.
https://blog.riseml.com/comparing-google-tpuv2-against-nvidia-v100-on-resnet-50-c2bbb6a51e5ehttps://medium.com/syncedreview/googles-tpu-chip-goes-public-in-challenge-to-nvidia-s-gpu-78ced56776b5https://sebastianraschka.com/Articles/2014_multiprocessing.htmlhttps://towardsdatascience.com/how-i-learned-to-love-parallelized-applies-with-python-pandas-dask-and-numba-f06b0b367138https://www.geeksforgeeks.org/multiprocessing-python-set-2/https://www.geeksforgeeks.org/multithreading-in-python-set-2-synchronization/https://medium.com/idealo-tech-blog/parallelisation-in-python-an-alternative-approach-b2749b49a1ehttps://stackoverflow.com/questions/990102/python-global-interpreter-lock-gil-workaround-on-multi-core-systems-using-taskhttps://stackoverflow.com/questions/38666078/fast-queue-of-read-only-numpy-arrayshttps://medium.com/@rvprasad/data-and-chunk-sizes-matter-when-using-multiprocessing-pool-map-in-python-5023c96875efhttps://stackabuse.com/parallel-processing-in-python/
https://blog.riseml.com/comparing-google-tpuv2-against-nvidia-v100-on-resnet-50-c2bbb6a51e5e
https://medium.com/syncedreview/googles-tpu-chip-goes-public-in-challenge-to-nvidia-s-gpu-78ced56776b5
https://sebastianraschka.com/Articles/2014_multiprocessing.html
https://towardsdatascience.com/how-i-learned-to-love-parallelized-applies-with-python-pandas-dask-and-numba-f06b0b367138
https://www.geeksforgeeks.org/multiprocessing-python-set-2/
https://www.geeksforgeeks.org/multithreading-in-python-set-2-synchronization/
https://medium.com/idealo-tech-blog/parallelisation-in-python-an-alternative-approach-b2749b49a1e
https://stackoverflow.com/questions/990102/python-global-interpreter-lock-gil-workaround-on-multi-core-systems-using-task
https://stackoverflow.com/questions/38666078/fast-queue-of-read-only-numpy-arrays
https://medium.com/@rvprasad/data-and-chunk-sizes-matter-when-using-multiprocessing-pool-map-in-python-5023c96875ef
https://stackabuse.com/parallel-processing-in-python/
a) Pool and Process:
https://docs.python.org/3/library/multiprocessing.htmlhttps://www.ellicium.com/python-multiprocessing-pool-process/
https://docs.python.org/3/library/multiprocessing.html
https://www.ellicium.com/python-multiprocessing-pool-process/
b) Threading:
3. https://realpython.com/python-gil/
4. https://stackoverflow.com/questions/29270818/why-is-a-python-i-o-bound-task-not-blocked-by-the-gil
5. https://stackoverflow.com/questions/27455155/python-multiprocessing-combined-with-multithreading
6. https://stackoverflow.com/questions/6893968/how-to-get-the-return-value-from-a-thread-in-python
7. https://stackoverflow.com/questions/200469/what-is-the-difference-between-a-process-and-a-thread
c) Dask
8. https://ml.dask.org
9. https://docs.dask.org/en/latest/
d) torch.multiprocessing:
9. https://pytorch.org/docs/stable/multiprocessing.html
10. https://pytorch.org/docs/stable/notes/multiprocessing.html
d) Pycuda:
11. https://documen.tician.de/pycuda/tutorial.html
12. https://github.com/inducer/pycuda/tree/master/examples
13. https://www3.nd.edu/~zxu2/acms60212-40212-S12/Lec-12-02.pdf
Suggestions and reviews are welcome.Thank you for reading!
Signed:
Some rights reserved