Title: Python Multiprocessing Pool (Concurent Futures ProcessPoolExecutor) slow down with increasing number of workers
Tags: python,multiprocessing,python-multiprocessing

Question: Problem description
Hi, I've got a computationally heavy function which I am running in parallel.
I've noticed that when using the concurrent futures ProcessPoolExecutor (or the multiprocessing Pool) the processes slows down when using more workers (or adding more tasks). I.e. when I run my code on 2 workers, the average execution time of a process is around 0.7s, however when using 16 workers (the cpu_count of my processor) the average execution time of the same process is of 6.7s.
A similar thing happens when running the same calculation on more tasks.
See the test code and results below.
Test code
import concurrent.futures
import os
from time import perf_counter

import numpy as np


def func(foo):
    start_time = perf_counter()
    long_calculation = np.random.random(size=100000000).std()
    stop_time = perf_counter()
    execution_time = stop_time - start_time
    return execution_time


cpu_count = os.cpu_count()
assert cpu_count is not None
print(f"CPU Count: {cpu_count}")


# ======= Increasing Tasks ============
# max_workers = 10
# for tasks in [1, 2, 5, 10, 20, 50, 100, 1000]:

# ======= Increasing Max Workers ============
tasks = 50
for max_workers in range(1, cpu_count + 1):
    with concurrent.futures.ProcessPoolExecutor(max_workers) as pool:
        total_exec_time: float = 0.0
        processes = pool.map(func, range(tasks))
        for process_result in processes:
            total_exec_time += process_result
        print(
            f"{tasks} tasks on {max_workers} workers - average process execution time:  {round(total_exec_time / tasks, 2)}s"
        )

The results
Increasing the number of max_workers
PU Count: 16
50 tasks on 1 workers - average process execution time:  0.65s
50 tasks on 2 workers - average process execution time:  0.76s
50 tasks on 3 workers - average process execution time:  0.97s
50 tasks on 4 workers - average process execution time:  1.16s
50 tasks on 5 workers - average process execution time:  1.6s
50 tasks on 6 workers - average process execution time:  1.9s
50 tasks on 7 workers - average process execution time:  2.29s
50 tasks on 8 workers - average process execution time:  2.67s
50 tasks on 9 workers - average process execution time:  3.02s
50 tasks on 10 workers - average process execution time:  3.5s
50 tasks on 11 workers - average process execution time:  4.0s
50 tasks on 12 workers - average process execution time:  4.73s
50 tasks on 13 workers - average process execution time:  5.37s
50 tasks on 14 workers - average process execution time:  5.66s
50 tasks on 15 workers - average process execution time:  6.07s
50 tasks on 16 workers - average process execution time:  6.71s

Increasing the number of tasks
10 workers with 1 tasks - average process execution time:  0.67s
10 workers with 2 tasks - average process execution time:  0.84s
10 workers with 5 tasks - average process execution time:  1.91s
10 workers with 10 tasks - average process execution time:  3.82s
10 workers with 20 tasks - average process execution time:  3.52s
10 workers with 50 tasks - average process execution time:  3.88s
10 workers with 100 tasks - average process execution time:  4.0s
10 workers with 1000 tasks - average process execution time:  3.92s

Results
Overall, the whole program runs faster, however it is slowing down progressively. I would like it to keep the same efficiency, no matter how many workers I use.

NOTE
The CPU usage is increasing linearly alongside the max_workers parameter.
The CPU usage is at 100% when using 15 or 16 max_workers (the idle CPU usage is around 7%) and my RAM has always plenty of free space.

Using multiprocessing
Side note: I get the exact same results using the multiprocessing Pool.
from multiprocessing import Pool

...
    with Pool(processes=max_workers) as pool:
        ...

Answer: Overall, the whole program runs faster, however it is slowing down progressively. I would like it to keep the same efficiency, no matter how many workers I use.

in short you can't always do that, each process is assigned a core with only around 5 MB of cache, if your process tries to use more memory than that it has to use the RAM, which has a limited bandwidth and is the reason of the bottleneck here, python alone takes much more than 5MB for its internal calls and will almost always be contending on the RAM bandwidth, there are also other bottlenecks like the OS scheduler and context switching.
lastly there is thermal throttling which happens when a CPU gets hotter, so you need a second CPU to spread the heat generation.
if you want the same speed when you increase the number of workers you have to increase the number of RAM chips and CPUs, which ultimately means, use cluster computing.
there are also faster methods like not storing the random numbers in memory and consuming them on construction which can eliminate this contention.
also using a language that runs on bare-metal like C++ will reduce the memory (and power) consumption which reduces this contention. (if you are not storing millions of numbers in memory).