Repository: numba/numba

Issue:
Memory leak when using parallel=True and slicing array
- [ ] I am using the latest released version of Numba (most recent is visible in
 the change log (https://github.com/numba/numba/blob/master/CHANGE_LOG).
I'm using 0.43.1 and can't seem to update to 0.44.1 in Anaconda?
- [X] I have included below a minimal working reproducer (if you are unsure how
 to write one see http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports).

For <reasons>, I'm doing some heavy computation that requires parallel and slicing an array.
Define function like so:

```import numpy as np
import numba

@numba.njit(parallel=True, fastmath=True)
def mean_matrix(arr):
	""" xs is a list of array

	"""

	c = 0.
	n = len(arr)
	for i in range(n):
		c += arr[i].sum()

	return c
```

Then in ipython, run the following:

```
arr = np.zeros((10000, 100, 100))

for i in range(1000):
	mean_matrix(arr)
```
And watch memory usage grow in task manager. If I take out `parallel=True`, this doesn't happen. Seems like sliced array isn't getting de-allocated when running in parallel.

Any ideas?


Discussion:
Thanks for the report. I can reproduce:
```python
import psutil
import numba
import numpy as np
rtsys = numba.runtime.rtsys

@numba.njit(parallel=True, fastmath=True)
def mean_matrix(arr):
    c = 0
    n = len(arr)
    for i in range(n):
            c += arr[i].sum()
    return c

arr = np.zeros((10000, 100, 100))

proc = psutil.Process()

print(proc.memory_full_info())

for i in range(10):
    print(("%s" % i).center(80, '-'))
    print(proc.memory_full_info())
    print(rtsys.get_allocation_stats())
    mean_matrix(arr)
```
this demonstrates the NRT (Numba Runtime) leaking, there's one more allocation than there is free, per run of the function, these accumulate.
CC: @DrTodd13 
Thanks for your prompt reply. So far I've sidestepped the bug by doing the slicing in another function.

```
@numba.njit()
def arr_sum(arr, i):
	return arr[i].sum()

@numba.njit(parallel=True, fastmath=True)
def mean_matrix(arr):
    c = 0
    n = len(arr)
    for i in range(n):
            c += arr_sum(arr, i)
    return c
```
@stevenwong thanks for updating this. This work around adds another hint as to where the problem lies!

RE:
> I'm using 0.43.1 and can't seem to update to 0.44.1 in Anaconda?

Which OS are you using? What's the error/reason given for refusing to update? Thanks.
@stuartarchibald I'm using Windows 10 with Anaconda 2019.03.
Google suggests that `numba` depends on `llvmlite` which depends on `distutils` which is part of `setuptools`.
The `--trusted-host` is required to get around a proxy issue at my company.

```
(base) C:\Users\swong>pip install setuptools --trusted-host pypi.python.org --trusted-host pypi.org --trusted-host files.pythonhosted.org --upgrade
Collecting setuptools
  Downloading https://files.pythonhosted.org/packages/ec/51/f45cea425fd5cb0b0380f5b0f048ebc1da5b417e48d304838c02d6288a1e/setuptools-41.0.1-py2.py3-none-any.whl (575kB)
    100% |████████████████████████████████| 583kB 6.6MB/s
Installing collected packages: setuptools
  Found existing installation: setuptools 40.8.0
    Uninstalling setuptools-40.8.0:
      Successfully uninstalled setuptools-40.8.0
Successfully installed setuptools-41.0.1
```
So that worked fine. Now when I update `numba`.
```
(base) C:\Users\swong>pip install numba --trusted-host pypi.python.org --trusted-host pypi.org --trusted-host files.pythonhosted.org --upgrade
Collecting numba
  Downloading https://files.pythonhosted.org/packages/1c/f7/626d5498c055d1d5853ae7a768080cdfc6852a7d5c66bdfde07ec7e84efa/numba-0.44.1-cp37-cp37m-win_amd64.whl (1.8MB)
    100% |████████████████████████████████| 1.9MB 819kB/s
Requirement already satisfied, skipping upgrade: numpy in c:\program files\anaconda3\lib\site-packages (from numba) (1.16.2)
Collecting llvmlite>=0.29.0 (from numba)
  Downloading https://files.pythonhosted.org/packages/e2/10/17af4ee132fd14fa17227a43f97930277eb72b97a189bbcec747588bfbdf/llvmlite-0.29.0-cp37-cp37m-win_amd64.whl (13.6MB)
    100% |████████████████████████████████| 13.6MB 595kB/s
Installing collected packages: llvmlite, numba
  Found existing installation: llvmlite 0.28.0
Cannot uninstall 'llvmlite'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.
```
Or if I try with conda:
```
(base) C:\Users\swong>conda update numba
WARNING conda.base.context:use_only_tar_bz2(632): Conda is constrained to only using the old .tar.bz2 file format because you have conda-build installed, and it is <3.18.3.  Update or remove conda-build to get smaller downloads and faster extractions.
Collecting package metadata (repodata.json): done
Solving environment: done

# All requested packages already installed.
```
The problem seems to just come from a `arr[i].sum()`.

Reproducer:

```python
import psutil
import numba
import numpy as np
rtsys = numba.runtime.rtsys

@numba.njit(parallel=True)
def mean_matrix(arr):
    return arr[i].sum()

arr = np.zeros((10000, 100, 100))

proc = psutil.Process()

print(proc.memory_full_info())

for i in range(10):
    print(("%s" % i).center(80, '-'))
    print(proc.memory_full_info())
    print(rtsys.get_allocation_stats())
    mean_matrix(arr)
```

There's a 1 ref leak per call.


Resolving patch:
diff --git a/numba/npyufunc/parfor.py b/numba/npyufunc/parfor.py
--- a/numba/npyufunc/parfor.py
+++ b/numba/npyufunc/parfor.py
@@ -381,6 +381,9 @@ def _lower_parfor_parallel(lowerer, parfor):
                             lowerer.fndesc.calltypes[rhs] = ct
                     lowerer.lower_inst(inst)
 
+        # Cleanup reduction variable
+        for v in redarrs.values():
+            lowerer.lower_inst(ir.Del(v.name, loc=loc))
     # Restore the original typemap of the function that was replaced temporarily at the
     # Beginning of this function.
     lowerer.fndesc.typemap = orig_typemap


Test patch:
diff --git a/numba/tests/test_parfors.py b/numba/tests/test_parfors.py
--- a/numba/tests/test_parfors.py
+++ b/numba/tests/test_parfors.py
@@ -20,7 +20,7 @@
 
 import numba
 from numba import unittest_support as unittest
-from .support import TestCase, captured_stdout
+from .support import TestCase, captured_stdout, MemoryLeakMixin
 from numba import njit, prange, stencil, inline_closurecall
 from numba import compiler, typing, errors
 from numba.targets import cpu
@@ -1467,6 +1467,36 @@ def test_impl(a, b):
         msg = ("The reshape API may only include one negative argument.")
         self.assertIn(msg, str(raised.exception))
 
+
+class TestParforsLeaks(MemoryLeakMixin, TestParforsBase):
+    def check(self, pyfunc, *args, **kwargs):
+        cfunc, cpfunc = self.compile_all(pyfunc, *args)
+        self.check_parfors_vs_others(pyfunc, cfunc, cpfunc, *args, **kwargs)
+
+    @skip_unsupported
+    def test_reduction(self):
+        # issue4299
+        @njit(parallel=True)
+        def test_impl(arr):
+            return arr.sum()
+
+        arr = np.arange(10).astype(np.float64)
+        self.check(test_impl, arr)
+
+    @skip_unsupported
+    def test_multiple_reduction_vars(self):
+        @njit(parallel=True)
+        def test_impl(arr):
+            a = 0.
+            b = 1.
+            for i in prange(arr.size):
+                a += arr[i]
+                b += 1. / (arr[i] + 1)
+            return a * b
+        arr = np.arange(10).astype(np.float64)
+        self.check(test_impl, arr)
+
+
 class TestPrangeBase(TestParforsBase):
 
     def __init__(self, *args):