Question: I have a Python script to process one single file (~100 Gb) with many simple statistical operation that do not depend on each other. So I first load the file, then I use Joblib to run the statistical operations on the loaded data in parallel. I can see from htop that all CPU cores are loaded to 100% but most of the time the load is red, which (according to htop) indicates kernel operations. To give more details of my implementation, I first load the file into one single numpy array defined as a global variable. Then I define the statistical operations as a function that uses that global variable, and finally with Joblib I run the functions. I think the red CPU load happens because all the parallel operations use the same variable. So I am asking if there is better way to do this. Below is a simplified minimal example of my implementation. myVariable = np.load("myFile.npy") myParameters = np.arange(100) def myFunction(myParameter): return np.mean(myVariable**myParameter) Parallel(n_jobs=-1)(delayed(myFunction)(p) for p in myParameters)  Please note that I have posted a similar question that I am still working on. In the other question the problem came from reading many files in parallel, and in that case the CPUs were not loaded and were always in D state. In this case only one file is read and many processes are executed using the loaded data, and in this case the CPUs are loaded but they shown in red.

Answer: TL;DR: there are several issues in this code making is very inefficient. Put it shortly, it is bound by system overheads like page faults, cache misses, cache-line write allocates, useless memory writes, and even possibly memory swapping (depending on the amount of available RAM).  Under the hood To understand why this code is inefficient, what are the overheads and how to fix them, we first need to understand what each line does and how modern systems actually work. First of all, np.load("myFile.npy") allocates a huge array of 100 Gb (that is 12.5 GB) and fill its content from a storage device. This part can be particularly slow using a HDD (at least few minutes), but it should be relatively fast with a SSD (at least few seconds). The speed of this part is dependent of the hardware, the OS modules/drivers, the OS itself and the way Numpy read data. Regarding myVariable**myParameter, it first allocates a new temporary array of the same size than myVariable. At this point, the OS does not reserve the space in RAM. It only reserve the space in virtual memory. More specifically, a group of virtual memory (fixed-size) chunks called pages (typically 4KB). This array is then filled with the result of myVariable[i]**myParameter for every i. When a value is written in a page for the first time (i.e. first touch), the OS maps the virtual memory page to a physical memory page in RAM. This operation is called a page fault and it is very expensive. In addition, big temporary arrays are slow to write in RAM on x86-64 systems because of the way CPU caches works (more specifically the cache coherence protocol). CPU caches are split in small memory chunks called cache lines. x86-64 CPUs (as well as some other architectures) requires a cache line to be read if a part of it is written, which is the case here. This is called cache line write allocations. This means half the bandwidth is wasted due to reads when a big temporary array is written. np.mean read the temporary array recently written in RAM so to produce a scalar value. Parallel(n_jobs=-1)(delayed(myFunction)(p) for p in myParameters) creates N processes, executing the target function. N is typically the number of (physical) cores, or even SMT threads (also called logical cores) on some machines. Global variables are serialised (using pickle) which is very slow due to interprocess communication (and pickling). However, based on this answer, Numpy arrays are shared between processes thanks to virtual memory and a smart implementation of JobLib. Each process one creates temporary arrays and compute the mean of it. This means the amount of RAM needed is at least N+1 times the size of myVariable. For a machine with 8 cores, this means 12.5*9 = 212.5 GB of RAM which is a huge amount of RAM. If there is not enough memory available, the OS need to store memory pages on a storage device area called the memory swap which is generally extremely slow compared to the speed of the RAM. One need to ensure there is enough memory to do the computation. In fact, one need to avoid allocating so much memory in the first place if possible. The cache and the RAM are used very inefficiently. Indeed, myVariable is read N times (because of the N processes) from the RAM while it can theoretically be read only once. The temporary arrays causes a lot of expensive cache misses due to the large amount of data written. They can make the computation even slower due to an effect called cache trashing. Note that the RAM of modern architectures tends to be so slow that only few core are often able to saturate it. For exemple, 2 cores of my i5-9600KF CPU (having 6-core) can saturate my 40 GiB/s RAM (2 channels of DDR4 RAM running at 3200 MHz).  How to write a faster code The first thing to to is to avoid creating huge temporary arrays like the plague. You can perform the computation using small chunks and reuse small temporary arrays for multiple chunks. Alternatively, you can use tools like Numba or Cython so to compute data on-the-fly preventing the need to creates temporary arrays here. This can be done using basic loops operating on scalar items. Loops are not slow with Numba or Cython because they are compiled to native code (if written correctly). This method can be combined with the previous one. This removes the overhead of page-faults, the RAM and the CPU caches are far more efficiently used (by not reloading data N times and not writing huge amount of data), not to mention it allocates far less memory avoiding memory swap issues. The benefits are so huge that 1 process running an optimised code can be significantly faster than your current code using N processes. The best actual strategy to use is dependent of your exact needs. For example, Numba cannot compile any arbitrary Python code : the code needs not to use modules other than Numpy yet (mainly). Cython is good to make loops faster but it cannot optimize Numpy function calls. Chunks-based code using Numpy is often not as fast as using Numba or Cython because of the overheads of Numpy. However, it is sometimes more flexible in practice. Besides, It might also be fast enough for your needs. In practice, Numba does not support compiling functions using large global variable efficiently (like in your case). But, this is not really a problem. We can add an indirection layer so to do that. Here is an example: import numba as nb # Caching is used to avoid the function to be recompiled in each process @nb.njit(cache=True) def myFunction_numba_seq(myParameter, myVariable): sum = 0.0 for i in range(myVariable.size): sum += myVariable[i] ** myParameter return sum / myVariable.size def myFunction(myParameter): return myFunction_numba_seq(myParameter, myVariable) # [...] Same code using Parallel(n_jobs=-1) here  That being said, using global variable is generally strongly discouraged in software engineering (e.g. for optimizations and for maintainability). Besides, Numba and Cython support multi-threading (without GIL issues). Thus, you can just use Numba without joblib in the following way: import numba as nb @nb.njit(parallel=True) def myFunction_numba_par(myParameters, myVariable): sums = np.zeros(myParameters.size) # Parallel loop. for i in nb.prange(myVariable.size): for j in range(myParameters.size): sums[j] += myVariable[i] ** myParameters[j] return sums / myVariable.size myVariable = np.load("myFile.npy") myParameters = np.arange(100) result = myFunction_numba_par(myParameters, myVariable)  Note that each item of myVariable (read once from RAM) is reused so to compute all the mean simultaneously. This means the code only need to read myVariable once. Note that this code can be improved using chunks to make it faster. Indeed, compilers can generate a fast SIMD instructions on some platforms. There are also fast math libraries to compute a**b often more efficiently than the default math library provided by the system. Also please note that summing very large arrays tends to cause accuracy issues. Chunks can help to make results more accurate here. There are also accurate algorithm to compute a mean (e.g. Kahan summation). Writing a fast accurate mean is actually not trivial to do. Numpy can does it quite well, so it can be simpler with chunks.