Repository: numba/numba

Issue:
Compilation with `parallel=True` gives garbage results for a very simply parallelizable function
<!--

Thanks for opening an issue! To help the Numba team handle your information
efficiently, please first ensure that there is no other issue present that
already describes the issue you have
(search at https://github.com/numba/numba/issues?&q=is%3Aissue).

-->

I have a relatively simple (and seemingly easily parallelizable) function for which I get different results when compiling with `parallel=True` and `parallel=False`. I guess that there must be some race conditions at play, but I really cannot see why.

I am not 100% sure whether it is a bug or just some fault in my code, but it was suggested to me on [SO](https://stackoverflow.com/questions/69958049/numba-parallel-true-produces-corrupted-results) that this might be a bug.

Below is a reproducible example.

```python
import numpy as np
from numba import njit, prange

def ubcm_jacobian(
    D: np.ndarray,
    b: np.ndarray
) -> np.ndarray:
    """Jacobian of UBCM negative log-likelihood.

    Parameters
    ----------
    D
        Degree sequence array.
    b
        Model parameters.
    """
    B = np.empty_like(b)
    b = np.exp(b)

    for i in prange(len(b)):
        bi   = b[i]
        x    = bi * b
        B[i] = (x / (1 + x)).sum()

    b2 = b**2
    B -= b2 / (1 + b2)

    return B - D


ubcm_not_parallel = njit(ubcm_jacobian, boundscheck=False, nogil=True, parallel=False)
ubcm_parallel = njit(ubcm_jacobian, boundscheck=False, nogil=True, parallel=True)

# Sample data
D = np.arange(1, 11)
b = np.log(D)

# Results
x1 = ubcm_not_parallel(D, b)
x2 = ubcm_parallel(D, b)

assert np.allclose(x1, x2), "results are not equal"
```

# Expected behavior

Of course the expected behavior is that the function should return identical results (up to the floating point precision) in both cases.



Discussion:
Thanks for the report, I can reproduce. MWR:
```python
import numpy as np
from numba import njit, prange

def foo(x):
    A = np.zeros_like(x)
    tmp = np.cos(x) # this can be any 1-arity ufunc

    for i in prange(len(x)):
        A[i] = tmp.sum()

    return A

foo_serial = njit(foo)
foo_parallel = njit(foo, parallel=True)

x = np.arange(10.)

x1 = foo(x)
x2 = foo_serial(x)
x3 = foo_parallel(x)

print(f"Python:\n{x1}")
print(f"Serial:\n{x2}")
print(f"Parallel:\n{x3}")

assert np.allclose(x1, x2), "results are not equal, serial"
assert np.allclose(x1, x3), "results are not equal, parallel"
```
gives:
```
$ python issue7578.py 
Python:
[0.42162378 0.42162378 0.42162378 0.42162378 0.42162378 0.42162378
 0.42162378 0.42162378 0.42162378 0.42162378]
Serial:
[0.42162378 0.42162378 0.42162378 0.42162378 0.42162378 0.42162378
 0.42162378 0.42162378 0.42162378 0.42162378]
Parallel:
[-1.22680415e+58 -1.09049258e+58 -9.54181004e+57 -8.17869432e+57
 -6.81557860e+57 -5.45246288e+57 -4.08934716e+57 -2.72623144e+57
 -1.36311572e+57  4.21623783e-01]
Traceback (most recent call last):
  File "issue7578.py", line 27, in <module>
    assert np.allclose(x1, x3), "results are not equal, parallel"
AssertionError: results are not equal, parallel
```
CC @DrTodd13 
Thanks for looking at this so quickly! I have one more remark (just in case it matters).

In my function the out-of-loop array `B` which is populated with values by the main loop is initialized to _empty_ values. In your reprex it is initialized to zeros. This probably does not matter for testing the problem, but I am mentioning this in case it may matter. And in practice this is sometimes important as initializing _empty_ arrays is much faster than zero arrays.
> Thanks for looking at this so quickly! I have one more remark (just in case it matters).
> 
> In my function the out-of-loop array `B` which is populated with values by the main loop is initialized to _empty_ values. In your reprex it is initialized to zeros. This probably does not matter for testing the problem, but I am mentioning this in case it may matter. And in practice this is sometimes important as initializing _empty_ arrays is much faster than zero arrays.

Thanks for the note. I made it zeros to flush the memory to try and see if it was junk from the original uninitialised memory or junk from something else. Indeed, for performance, don't allocate zeros unless you _really_ need to!
Also, @DrTodd13, I think this is an issue with loop fusion, the `np.cos()` loop is being fused with the `prange`, but the `prange` body has a reduction on the result. Adding in a function call to prevent fusion leads to the MWR working:
```python
import numpy as np
from numba import njit, prange

@njit
def sink():
    pass

def foo(x):
    A = np.zeros_like(x)
    tmp = np.cos(x) # this can be any 1-arity ufunc
    sink()
    for i in prange(len(x)):
        A[i] = tmp.sum()

    return A

foo_serial = njit(foo)
foo_parallel = njit(foo, parallel=True)

x = np.arange(10.)

x1 = foo(x)
x2 = foo_serial(x)
x3 = foo_parallel(x)

print(f"Python:\n{x1}")
print(f"Serial:\n{x2}")
print(f"Parallel:\n{x3}")

print(foo_parallel.parallel_diagnostics(level=4))

assert np.allclose(x1, x2), "results are not equal, serial"
assert np.allclose(x1, x3), "results are not equal, parallel"
```

Resolving patch:
diff --git a/numba/core/ir_utils.py b/numba/core/ir_utils.py
--- a/numba/core/ir_utils.py
+++ b/numba/core/ir_utils.py
@@ -744,6 +744,7 @@ def has_no_side_effect(rhs, lives, call_table):
             call_list == [array_analysis.wrap_index] or
             call_list == [prange] or
             call_list == ['prange', numba] or
+            call_list == ['pndindex', numba] or
             call_list == [parfor.internal_prange] or
             call_list == ['ceil', math] or
             call_list == [max] or
diff --git a/numba/parfors/parfor.py b/numba/parfors/parfor.py
--- a/numba/parfors/parfor.py
+++ b/numba/parfors/parfor.py
@@ -598,6 +598,9 @@ def __repr__(self):
         return "id=" + str(self.id) + repr(self.loop_nests) + \
             repr(self.loop_body) + repr(self.index_var)
 
+    def get_loop_nest_vars(self):
+        return [x.index_variable for x in self.loop_nests]
+
     def list_vars(self):
         """list variables used (read/written) in this parfor by
         traversing the body and combining block uses.
@@ -3563,6 +3566,74 @@ def _find_parfors(body):
             yield i, inst
 
 
+def _is_indirect_index(func_ir, index, nest_indices):
+    index_def = guard(get_definition, func_ir, index.name)
+    if isinstance(index_def, ir.Expr) and index_def.op == 'build_tuple':
+        if [x.name for x in index_def.items] == [x.name for x in nest_indices]:
+            return True
+
+    return False
+
+
+def get_array_indexed_with_parfor_index_internal(loop_body,
+                                                 index,
+                                                 ret_indexed,
+                                                 ret_not_indexed,
+                                                 nest_indices,
+                                                 func_ir):
+    for blk in loop_body:
+        for stmt in blk.body:
+            if isinstance(stmt, (ir.StaticSetItem, ir.SetItem)):
+                setarray_index = get_index_var(stmt)
+                if (isinstance(setarray_index, ir.Var) and
+                    (setarray_index.name == index or
+                     _is_indirect_index(
+                         func_ir,
+                         setarray_index,
+                         nest_indices))):
+                    ret_indexed.add(stmt.target.name)
+                else:
+                    ret_not_indexed.add(stmt.target.name)
+            elif (isinstance(stmt, ir.Assign) and
+                  isinstance(stmt.value, ir.Expr) and
+                  stmt.value.op in ['getitem', 'static_getitem']):
+                getarray_index = stmt.value.index
+                getarray_name = stmt.value.value.name
+                if (isinstance(getarray_index, ir.Var) and
+                    (getarray_index.name == index or
+                     _is_indirect_index(
+                         func_ir,
+                         getarray_index,
+                         nest_indices))):
+                    ret_indexed.add(getarray_name)
+                else:
+                    ret_not_indexed.add(getarray_name)
+            elif isinstance(stmt, Parfor):
+                get_array_indexed_with_parfor_index_internal(
+                    stmt.loop_body.values(),
+                    index,
+                    ret_indexed,
+                    ret_not_indexed,
+                    nest_indices,
+                    func_ir)
+
+
+def get_array_indexed_with_parfor_index(loop_body,
+                                        index,
+                                        nest_indices,
+                                        func_ir):
+    ret_indexed = set()
+    ret_not_indexed = set()
+    get_array_indexed_with_parfor_index_internal(
+        loop_body,
+        index,
+        ret_indexed,
+        ret_not_indexed,
+        nest_indices,
+        func_ir)
+    return ret_indexed, ret_not_indexed
+
+
 def get_parfor_outputs(parfor, parfor_params):
     """get arrays that are written to inside the parfor and need to be passed
     as parameters to gufunc.
@@ -4136,6 +4207,7 @@ def get_user_varname(v):
     p1_body_defs = set()
     for defs in p1_body_usedefs.defmap.values():
         p1_body_defs |= defs
+    p1_body_defs |= get_parfor_writes(parfor1)
     # Add reduction variables from parfor1 to the set of body defs
     # so that if parfor2 reads the reduction variable it won't fuse.
     p1_body_defs |= set(parfor1.redvars)
@@ -4145,13 +4217,25 @@ def get_user_varname(v):
     for uses in p2_usedefs.usemap.values():
         p2_uses |= uses
 
-    if not p1_body_defs.isdisjoint(p2_uses):
-        dprint("try_fuse: parfor2 depends on parfor1 body")
-        msg = ("- fusion failed: parallel loop %s has a dependency on the "
-                "body of parallel loop %s. ")
-        report = FusionReport(parfor1.id, parfor2.id,
-                                msg % (parfor1.id, parfor2.id))
-        return None, report
+    overlap = p1_body_defs.intersection(p2_uses)
+    # overlap are those variable defined in first parfor and used in the second
+    if len(overlap) != 0:
+        # Get all the arrays
+        _, p2arraynotindexed = get_array_indexed_with_parfor_index(
+            parfor2.loop_body.values(),
+            parfor2.index_var.name,
+            parfor2.get_loop_nest_vars(),
+            func_ir)
+
+        unsafe_var = (not isinstance(typemap[x], types.ArrayCompatible) or x in p2arraynotindexed for x in overlap)
+
+        if any(unsafe_var):
+            dprint("try_fuse: parfor2 depends on parfor1 body")
+            msg = ("- fusion failed: parallel loop %s has a dependency on the "
+                    "body of parallel loop %s. ")
+            report = FusionReport(parfor1.id, parfor2.id,
+                                    msg % (parfor1.id, parfor2.id))
+            return None, report
 
     return fuse_parfors_inner(parfor1, parfor2)
 


Test patch:
diff --git a/numba/tests/test_parfors.py b/numba/tests/test_parfors.py
--- a/numba/tests/test_parfors.py
+++ b/numba/tests/test_parfors.py
@@ -4321,6 +4321,19 @@ def test_issue_due_to_max_label(self):
         )
         self.assertIn("TEST PASSED", out.decode())
 
+    @skip_parfors_unsupported
+    def test_issue7578(self):
+        def test_impl(x):
+            A = np.zeros_like(x)
+            tmp = np.cos(x) # this can be any 1-arity ufunc
+
+            for i in range(len(x)):
+                A[i] = tmp.sum()
+
+            return A
+
+        x = np.arange(10.)
+        self.prange_tester(test_impl, x)
 
 @skip_parfors_unsupported
 class TestParforChunksizing(TestCase):