Repository: numba/numba

Issue:
parallel=True in nopython mode seems to make memory leak
## Reporting a bug

So I am doing a 'multiplication' of some sort. It works fine without 'parallel=True' and even though it gives me the right results while in parallel, it eats up my RAM quite quickly.


- macOS 12.4 (MacBook Pro 16 with M1 Pro) 
- Python 3.9.13
- numba 0.56.0, numpy 1.19.5, llvmlite 0.39.0


So the first function `multiplication` works perfectly fine, but as I am using it over many iterations in my full code, I need to make it much faster.
So I tried to make it run in parallel, but it seems to make a memory leak that makes my (previously working) program crash after some number of iterations.


Here is my simplified code:

``` Python3
import numpy as np
from numba import njit, get_num_threads, prange, float64
from numba.typed import List, Dict

@njit
def multiplication(left, right, operator, length):
    result_functional = np.zeros(shape=length, dtype=float64)
    for i, left_value in enumerate(left):
        for j, right_value in enumerate(right):
            for k, count in operator[i][j].items():
                result_functional[k] += count * left_value * right_value
    return result_functional

@njit(parallel=True)
def pmultiplication(left, right, operator, length):
    result_functional = np.zeros(shape=length, dtype=float64)
    num_loads = get_num_threads()
    load_size = len(left) / num_loads
    for n in prange(num_loads):
        result_functional += multiplication(left[round(n * load_size): round((n+1) * load_size)], right,
                                            operator[round(n * load_size): round((n+1) * load_size)], length)
    return result_functional

@njit
def random_operator(dim, trunc):
    side_length = np.sum(dim ** np.arange(trunc + 1))
    full_length = np.sum(dim ** np.arange(2*trunc + 1))
    numba_shuffle_operator = List()
    for i in range(side_length):
        numba_list = List()
        for j in range(side_length):
            num = np.random.randint(low=1, high=(i+j+2))
            keys = np.random.randint(low=0, high=full_length, size=num)
            values = np.random.randint(low=1, high=num+1, size=num)
            numba_dict = Dict()
            for k in range(num):
                numba_dict[keys[k]] = values[k]
            numba_list.append(numba_dict)
        numba_shuffle_operator.append(numba_list)
    return numba_shuffle_operator


def prepare_multiplication(dim, trunc):
    operator = random_operator(dim, trunc)
    left = np.random.random(size=len(operator))
    right = np.random.random(size=len(operator[0]))
    length = np.sum(dim ** np.arange(2 * trunc + 1))
    return left, right, operator, length

# # compilation
left, right, operator, length = prepare_multiplication(dim=2, trunc=2)
multiplication(left, right, operator, length)
pmultiplication(left, right, operator, length)

# # parameters that will be used
left, right, operator, length = prepare_multiplication(dim=3, trunc=4)

```

And here is my code to show how the memory is eaten up:

```Python3
import psutil
import pandas as pd
from IPython.display import display

num_iter = 100
process = psutil.Process()

cached_rss = np.array([process.memory_info().rss])
for i in range(num_iter):
    multiplication(left, right, operator, length)
    cached_rss = np.append(cached_rss, process.memory_info().rss)

cached_rss_parallel = np.array([process.memory_info().rss])
for i in range(num_iter):
    pmultiplication(left, right, operator, length)
    cached_rss_parallel = np.append(cached_rss_parallel, process.memory_info().rss)

memory = pd.DataFrame()
memory["diff since start"] = cached_rss - cached_rss[0]
memory["diff since last"] = 0
memory["diff since last"][1:] = np.diff(cached_rss, n=1)
memory["diff since start (parallel)"] = cached_rss_parallel - cached_rss_parallel[0]
memory["diff since last (parallel)"] = 0
memory["diff since last (parallel)"][1:] = np.diff(cached_rss_parallel, n=1)

pd.options.display.float_format = '{:,.0f}'.format
display(memory.tail(5).astype(float))
print(f'mean: {np.mean(memory["diff since last (parallel)"]):,.2f}')
print(f'standard dev: {np.std(memory["diff since last (parallel)"]):,.2f}')

```

<table border="1" class="dataframe">  <thead>    <tr style="text-align: right;">      <th></th>      <th>diff since start</th>      <th>diff since last</th>      <th>diff since start (parallel)</th>      <th>diff since last (parallel)</th>    </tr>  </thead>  <tbody>    <tr>      <th>96</th>      <td>0</td>      <td>0</td>      <td>3,342,336</td>      <td>98,304</td>    </tr>    <tr>      <th>97</th>      <td>0</td>      <td>0</td>      <td>3,342,336</td>      <td>0</td>    </tr>    <tr>      <th>98</th>      <td>0</td>      <td>0</td>      <td>3,440,640</td>      <td>98,304</td>    </tr>    <tr>      <th>99</th>      <td>0</td>      <td>0</td>      <td>3,440,640</td>      <td>0</td>    </tr>    <tr>      <th>100</th>      <td>0</td>      <td>0</td>      <td>3,522,560</td>      <td>81,920</td>    </tr>  </tbody></table>


mean: 34,876.83
standard dev: 86,869.42



Discussion:
Hi @louisamand, thanks for the report. 

I couldn't reproduce the crash. Which set of inputs does the crash happen? Also, the script you posted gives me the following output:
```
     diff since start  diff since last  diff since start (parallel)  diff since last (parallel)
96                  0                0                            0                           0
97                  0                0                            0                           0
98                  0                0                            0                           0
99                  0                0                            0                           0
100                 0                0                            0                           0
mean: 0.00
standard dev: 0.00
```

Hi @guilhermeleobas, thanks for the quick response ;)


I just restarted the jupyter kernel and the results were similar.  I also just tried it on a .py file (both with PyCharm and the Terminal) and it was no different. Computer restart does not seem to change anything either.

The true parameters for the full code are `dim=3` and `trunc=6`:
- The `operator` "matrix" is of dimension 1093 x 1093 and each "point" is a `numba.typed.Dict` with key type `int64` and value type `uint16` (with increasing number of elements as $i, j \to 1093$). 
The pure Python version of the matrix weights around 14 GiB.
- `length` is 797,161

I run a `scipy.optimize.minimize` algorithm on this "multiplication" and my Activity Monitor shows python3.9 uses more than 100 GiB before it finally crashes and the RAM clears.




Hi @louisamand, so, the program is crashing because it used all the RAM memory available?
Yeah I think it is the cause.
When not using it in parallel, there is no such problem and the full program runs for hours.
Thanks. I'll add it for discussion in the next triage meeting.

Resolving patch:
diff --git a/numba/parfors/parfor_lowering.py b/numba/parfors/parfor_lowering.py
--- a/numba/parfors/parfor_lowering.py
+++ b/numba/parfors/parfor_lowering.py
@@ -112,6 +112,7 @@ def _lower_parfor_parallel(lowerer, parfor):
     # init reduction array allocation here.
     nredvars = len(parfor_redvars)
     redarrs = {}
+    to_cleanup = []
     if nredvars > 0:
         # reduction arrays outer dimension equal to thread count
         scope = parfor.init_block.scope
@@ -210,6 +211,7 @@ def _lower_parfor_parallel(lowerer, parfor):
 
             # Remember mapping of original reduction array to the newly created per-worker reduction array.
             redarrs[redvar.name] = redarr_var
+            to_cleanup.append(redarr_var)
 
             init_val = parfor_reddict[red_name].init_val
 
@@ -244,6 +246,8 @@ def _lower_parfor_parallel(lowerer, parfor):
                         typ=redvar_typ,
                         name="redtoset",
                     )
+                    # rettoset is an array from np.full() and must be released
+                    to_cleanup.append(redtoset)
                 else:
                     redtoset = pfbdr.make_const_variable(
                         cval=init_val,
@@ -367,7 +371,7 @@ def _lower_parfor_parallel(lowerer, parfor):
         )
 
     # Cleanup reduction variable
-    for v in redarrs.values():
+    for v in to_cleanup:
         lowerer.lower_inst(ir.Del(v.name, loc=loc))
     # Restore the original typemap of the function that was replaced temporarily at the
     # Beginning of this function.


Test patch: