Repository: numba/numba

Issue:
Wrong results  when using jit(parallel=True)
This bug report is very similar to issue #7984 .
Using Python 3.8.7,  numba-0.55.1 with llvmlite-0.38.0, Under Win-10:


```
from numpy import ndarray
import numpy as np
import numba
from numba import jit


def test_par(img: ndarray) -> ndarray:

    rad = 5
    w2 = rad + rad

    min_value = np.amin(img)
    # print("")
    img = img - min_value
    s = img.shape
    a_p = np.zeros((s[0] + w2, s[1] + w2), dtype=np.int32)
    a_p[rad:rad+s[0], rad:rad+s[1]] = img

    b = np.zeros(s, dtype=np.float32)

    for i in range(rad):
        for j in range(rad):
            a_1 = a_p[i:i+s[0], j:j+s[1]]
            b = b + a_1
    
    return b

test_np = test_par
test_nb = numba.jit(nopython=True, fastmath=True, parallel=False)(test_par)
test_nb_p = numba.jit(nopython=True, fastmath=True, parallel=True)(test_par)

if __name__ == '__main__':
    image = np.empty((256, 256), dtype=np.uint8)
    for k in range(256):
        for m in range(256):
            image[k, m] = k

    res0 = test_np(image)
    res1 = test_nb(image)
    res2 = test_nb_p(image)

    diff1 = np.max(np.abs(res1-res0))
    print(diff1)
    diff2 = np.max(np.abs(res2-res0))
    print(diff2)

```

The results are:
0.0
4250.0

There seems to be a race condition that causes different results for` parallel=true.`
Strangely enough, when uncommenting the  `print("") `command, the results are OK.
Looks like the print command stops somehow the race condition (like openning the box of Schrödinger's cat:-)).

Tracking back the problem,  on numba-0.52.0 &  llvmlite-0.35.0 this problem does not exist,
but on  numba-0.53.0 &  llvmlite-0.36.0 it appears.

I hope that you can reproduce this bug.

Thanks,




Discussion:
Confirmed on MacOS with Python 3.10.2, Numba 0.55.1, ad NumPy 1.21.5. Though, I get a different number: 5100.0

Interestingly, that is the only place where doing `print("")` fixes the problem.
Thanks for confirming the bug.
Actually, on another win-10 machine, I got a different result (4800).
I replaced, `a_1 = a_p[i:i+s[0], j:j+s[1]]` with `a_1 = a_p[i, j]` and I got same results all the time. So, probably a problem with array slicing in parallel mode?

Additional points (will be updating),

1. I updated `get_thread_count` to return a specific number of threads. When it returns 1 to 4 (including both), results were same and equal to 0.0. However, from 5 onwards, I got different results.
Thanks for the report @AvinoamK. Thanks also for debugging this further @apmasell and @czgdp1807. That inserting a `print` in that specific location "fixes" it suggests it's probably an issue to do with fusion of parfors nodes as the `print` statement would defeat that.

Here's a MWR:
```python
from numba import njit
import numpy as np

parallel_options = {
    'comprehension': True,  # parallel comprehension
    'prange':        True,  # parallel for-loop
    'numpy':         True,  # parallel numpy calls
    'reduction':     True,  # parallel reduce calls
    'setitem':       True,  # parallel setitem
    'stencil':       True,  # parallel stencils
    'fusion':        True,  # enable fusion or not
}


@njit(parallel=parallel_options)
def foo(img):
    min_value = np.amin(img)
    img = img - min_value
    return img


n = 2
image = np.arange(n * n, dtype=np.uint8).reshape((n, n))

expected = foo.py_func(image.copy())
got = foo(image.copy())

print(foo.parallel_diagnostics())

np.testing.assert_allclose(expected, got)
```
if the `'fusion'` option in the `parallel` options dictionary is set to `False` then the code works as expected. Fusion probably shouldn't occur in this case as the fused code would be computing the `amin` of an array that's being mutated. Would guess the cross iteration dependency checks need making aware of this sort of use case, but would need to look at it more.
I investigated a little bit more after your comment @stuartarchibald . I think that the `has_cross_iter_dep` is giving a False positive that there is no cross iteration dependency. So, I applied the following diff which adds a conservative check to figure out cross iteration dependency. Basically after the diff, if in a loop body, two successive statements have a common variable then `has_cross_iter_dep` will assume that there is a cross iteration dependency. However I think this is very restrictive. May be we can relax it to, "only when LHS of the previous statement is the RHS of the current statement" then assume a cross iteration dependency. One more example where `main` fails but with the applied diff things workout.

https://github.com/numba/numba/blob/87ab859d95aa6322a72d3d31027f39a6ddf4af10/numba/parfors/parfor.py#L4113

**Diff**

```diff
diff --git a/numba/parfors/parfor.py b/numba/parfors/parfor.py
index c24a389e1..f9cc1e4b4 100644
--- a/numba/parfors/parfor.py
+++ b/numba/parfors/parfor.py
@@ -4117,9 +4117,11 @@ def has_cross_iter_dep(parfor):
     # TODO: make it more accurate using ud-chains
     indices = {l.index_variable for l in parfor.loop_nests}
     for b in parfor.loop_body.values():
+        stmts_vars = []
         for stmt in b.body:
             # GetItem/SetItem nodes are fine since can't have expression ins
ide
             # and only simple indices are possible
+            stmts_vars.append(stmt.list_vars())
             if isinstance(stmt, (ir.SetItem, ir.StaticSetItem)):
                 continue
             # tuples are immutable so no expression on parfor possible
@@ -4131,6 +4133,13 @@ def has_cross_iter_dep(parfor):
             if not indices.isdisjoint(stmt.list_vars()):
                 dprint("has_cross_iter_dep found", indices, stmt)
                 return True
+
+        for i in range(len(stmts_vars) - 1):
+            if (stmts_vars[i] is not None and
+                stmts_vars[i + 1] is not None and
+                not set(stmts_vars[i]).isdisjoint(set(stmts_vars[i + 1]))):
+                return True
+
     return False
 
 
```

**Definition of `foo` which fail with `main` branch but work with the above diff**

```python
@njit(parallel=parallel_options)
def foo(img1, img2):
    min_value1 = np.amin(img1)
    img2 = img1 - min_value1
    return img1, img2
```

Please feel free to correct me if I mis-interpreted something. I am learning about Numba with the help of these investigations and your comments. Thanks. :-)

Resolving patch:
diff --git a/numba/parfors/parfor.py b/numba/parfors/parfor.py
--- a/numba/parfors/parfor.py
+++ b/numba/parfors/parfor.py
@@ -2902,6 +2902,20 @@ def run(self):
             self.func_ir._definitions = build_definitions(self.func_ir.blocks)
             self.array_analysis.equiv_sets = dict()
             self.array_analysis.run(self.func_ir.blocks)
+
+            # Get parfor params to calculate reductions below.
+            _, parfors = get_parfor_params(self.func_ir.blocks,
+                                           self.options.fusion,
+                                           self.nested_fusion_info)
+
+            # Find reductions so that fusion can be disallowed if a
+            # subsequent parfor read a reduction variable.
+            for p in parfors:
+                p.redvars, p.reddict = get_parfor_reductions(self.func_ir,
+                                                             p,
+                                                             p.params,
+                                                             self.calltypes)
+
             # reorder statements to maximize fusion
             # push non-parfors down
             maximize_fusion(self.func_ir, self.func_ir.blocks, self.typemap,
@@ -2977,7 +2991,10 @@ def run(self):
 
             # Validate reduction in parfors.
             for p in parfors:
-                p.redvars, p.reddict = get_parfor_reductions(self.func_ir, p, p.params, self.calltypes)
+                p.redvars, p.reddict = get_parfor_reductions(self.func_ir,
+                                                             p,
+                                                             p.params,
+                                                             self.calltypes)
 
             # Validate parameters:
             for p in parfors:
@@ -4091,6 +4108,9 @@ def get_user_varname(v):
     p1_body_defs = set()
     for defs in p1_body_usedefs.defmap.values():
         p1_body_defs |= defs
+    # Add reduction variables from parfor1 to the set of body defs
+    # so that if parfor2 reads the reduction variable it won't fuse.
+    p1_body_defs |= set(parfor1.redvars)
 
     p2_usedefs = compute_use_defs(parfor2.loop_body)
     p2_uses = compute_use_defs({0: parfor2.init_block}).usemap[0]


Test patch:
diff --git a/numba/tests/test_parfors.py b/numba/tests/test_parfors.py
--- a/numba/tests/test_parfors.py
+++ b/numba/tests/test_parfors.py
@@ -1405,6 +1405,13 @@ def test_impl(A):
         # this doesn't fuse due to mixed indices
         self.assertEqual(countParfors(test_impl, (numba.float64[:,:],)), 2)
 
+        def test_impl(A):
+            min_val = np.amin(A)
+            return A - min_val
+        self.check(test_impl, A)
+        # this doesn't fuse due to use of reduction variable
+        self.assertEqual(countParfors(test_impl, (numba.float64[:],)), 2)
+
     def test_use_of_reduction_var1(self):
         def test_impl():
             acc = 0