Repository: numba/numba

Issue:
Bad interaction (hang) involving TBB and subprocess fork and maybe MKL OMP
This is documenting an issue discovered during 0.59.1. By writing it in the issue tracker, hopefully it can be searchable in case similar problems occur in the future for us or others. The issue being described here is already fixed by https://github.com/numba/numba/pull/9480/commits/b39bdf2516174c6f723d6785bd5f70e7a301b4cd#. (Self reminder: cherrypick the fix to main and link this this issue)

The `test_issue9490_non_det_ssa_problem` testcase can hang on very specifically Python 3.9, NumPy 1.25 and Linux x86-64 at this line: https://github.com/numba/numba/blob/147c641fb337c41029e9e8832d7cb40be700e413/numba/tests/parfor_iss9490_usecase.py#L71
 

GDB reveals output:

```
(gdb) info threads
  Id   Target Id                                    Frame
* 1    Thread 0x7ffff7c39280 (LWP 3954038) "python" __pthread_clockjoin_ex (threadid=140736834631424, thread_return=0x0,
    clockid=<optimized out>, abstime=<optimized out>, block=<optimized out>) at pthread_join_common.c:145
  2    Thread 0x7fffdc49c700 (LWP 3954042) "python" syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  3    Thread 0x7fffdc09b700 (LWP 3954043) "python" 0x00007ffff7d3f89b in sched_yield () at ../sysdeps/unix/syscall-template.S:78
  4    Thread 0x7fffdbc9a700 (LWP 3954044) "python" 0x00007ffff7d3f89b in sched_yield () at ../sysdeps/unix/syscall-template.S:78
  5    Thread 0x7fffdb899700 (LWP 3954045) "python" 0x00007ffff7d3f89b in sched_yield () at ../sysdeps/unix/syscall-template.S:78
  6    Thread 0x7fffdb498700 (LWP 3954047) "python" syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  7    Thread 0x7fffdac96700 (LWP 3954048) "python" 0x00007ffff7d3f89b in sched_yield () at ../sysdeps/unix/syscall-template.S:78
  8    Thread 0x7fffdb097700 (LWP 3954046) "python" 0x00007ffff7d3f89b in sched_yield () at ../sysdeps/unix/syscall-template.S:78
  12   Thread 0x7fffd9c92700 (LWP 3954054) "python" 0x00007ffff7d3f89b in sched_yield () at ../sysdeps/unix/syscall-template.S:78
  13   Thread 0x7fffd9490700 (LWP 3954053) "python" 0x00007ffff7d3f89b in sched_yield () at ../sysdeps/unix/syscall-template.S:78
  14   Thread 0x7fffda895700 (LWP 3954050) "python" 0x00007ffff7d3f89b in sched_yield () at ../sysdeps/unix/syscall-template.S:78
  15   Thread 0x7fffd908f700 (LWP 3954055) "python" 0x00007ffff5b4349a in __kmp_acquire_ticket_lock ()
   from /home/siu/miniconda3/envs/testenv_bad/lib/python3.9/site-packages/mkl/../../../libiomp5.so
```


```

Thread 1 "python" received signal SIGINT, Interrupt.
__pthread_clockjoin_ex (threadid=140736414611200, thread_return=0x0, clockid=<optimized out>, abstime=<optimized out>, block=<optimized out>) at pthread_join_common.c:145
145	pthread_join_common.c: No such file or directory.
(gdb) bt
#0  __pthread_clockjoin_ex (threadid=140736414611200, thread_return=0x0, clockid=<optimized out>, abstime=<optimized out>, block=<optimized out>)
    at pthread_join_common.c:145
#1  0x00007fffdcba294f in tbb::detail::r1::rml::internal::thread_monitor::join (
    handle=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>)
    at /usr/local/src/conda/tbb-2021.8.0/src/tbb/rml_thread_monitor.h:201
#2  tbb::detail::r1::rml::private_worker::release_handle (join=<optimized out>, handle=<optimized out>)
    at /usr/local/src/conda/tbb-2021.8.0/src/tbb/private_server.cpp:231
#3  tbb::detail::r1::rml::private_worker::start_shutdown (this=0x7fffdc7eb900) at /usr/local/src/conda/tbb-2021.8.0/src/tbb/private_server.cpp:256
#4  tbb::detail::r1::rml::private_worker::start_shutdown (this=0x7fffdc7eb900) at /usr/local/src/conda/tbb-2021.8.0/src/tbb/private_server.cpp:236
#5  tbb::detail::r1::rml::private_server::request_close_connection (this=0x7fffdcad3f80) at /usr/local/src/conda/tbb-2021.8.0/src/tbb/private_server.cpp:188
#6  0x00007fffdcb9c473 in tbb::detail::r1::market::release (this=0x7fffdcacf580, is_public=<optimized out>, blocking_terminate=<optimized out>)
    at /usr/local/src/conda/tbb-2021.8.0/src/tbb/market.cpp:214
#7  0x00007fffdcb9770a in tbb::detail::r1::finalize_impl (handle=...,
    handle@entry=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>)
    at /usr/local/src/conda/tbb-2021.8.0/src/tbb/governor.cpp:290
#8  tbb::detail::r1::finalize (mode=1, mode@entry=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, handle=...,
    handle@entry=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>)
    at /usr/local/src/conda/tbb-2021.8.0/src/tbb/governor.cpp:303
#9  tbb::detail::r1::finalize (handle=..., mode=1) at /usr/local/src/conda/tbb-2021.8.0/src/tbb/governor.cpp:298
#10 0x00007fffdcb7d1db in prepare_fork() ()
   from /home/siu/miniconda3/envs/testenv_bad/lib/python3.9/site-packages/numba/np/ufunc/tbbpool.cpython-39-x86_64-linux-gnu.so
#11 0x00007ffff7cd1c50 in __run_fork_handlers (who=<optimized out>, who@entry=atfork_run_prepare, do_locking=do_locking@entry=true) at register-atfork.c:123
#12 0x00007ffff7d20094 in __libc_fork () at ../sysdeps/nptl/fork.c:58
#13 0x00007ffff7fad47b in subprocess_fork_exec (self=<optimized out>, args=<optimized out>) at /usr/local/src/conda/python-3.9.18/Modules/_posixsubprocess.c:844
#14 0x00000000005073b0 in cfunction_call (func=0x7ffff6bce5e0, args=<optimized out>, kwargs=<optimized out>)
    at /usr/local/src/conda/python-3.9.18/Objects/methodobject.c:552
```


Important observations here:
1. Thread 1 is stuck at `__pthread_clockjoin_ex` as part of a `prepare_fork` call for the TBB backend. TBB is known to have problems during fork. This explains the hang. But it is surprising for `numba.testing.assert_allclose` would cause a fork.
2. Thread 14 is from OMP in MKL. It is not confirmed whether having OMP threadpool have an effect in this hang.

Using PDB to trace the hanging line, it is discovered that NumPy 1.25+ introduced this lines which call subprocess thus fork:

https://github.com/numpy/numpy/blob/maintenance/1.25.x/numpy/testing/_private/utils.py#L1239-L1253

Together with the lazy import by `numba.testing.assert_allclose`, it explains the use of fork. 

As to why the test is not failing with NumPy 1.26:

- first, NumPy 1.26 conda package does not support Python 3.9. Thus we only test Python 3.10+. 
- Python 3.10 has several changes to `subprocess` which may have averted bugs with fork; e.g. https://github.com/python/cpython/issues/91401

The [fix] (https://github.com/numba/numba/blob/147c641fb337c41029e9e8832d7cb40be700e413/numba/tests/parfor_iss9490_usecase.py#L71) is done by forcing one Numba thread thus one TBB thread. That perhaps avoided `subprocess` fork bug. I didn't do further analysis since the bug is fixed. 






Discussion:

Resolving patch:
diff --git a/maint/towncrier_checker.py b/maint/towncrier_checker.py
--- a/maint/towncrier_checker.py
+++ b/maint/towncrier_checker.py
@@ -66,16 +66,21 @@
 
     # Find PRs that are mentioned in CHANGE_LOG already
     # (most likely cherrypicked to previous bugfix releases)
-    with open(Path(os.path.dirname(__file__)) / ".." / "CHANGE_LOG") as fin:
-        # Use regex to find all PR numbers in the file and remove them
-        pr_re = re.compile(r"\#(\d+)")
-        prs_in_changelog = [
-            match.group(1) for match in pr_re.finditer(fin.read())
-        ]
-        to_remove = {pr for pr in prs if pr in prs_in_changelog}
-        for pr in to_remove:
-            print(f"Removing {pr} as already in CHANGELOG")
-            del prs[pr]
+    basepath = Path(os.path.dirname(__file__)) / ".."
+    oldchangelog = basepath / "CHANGE_LOG"
+    newchangelogs = (basepath / "docs" / "source" / "release").glob("*.rst")
+    for path in [oldchangelog, *newchangelogs]:
+        print('Checking', path)
+        with open(path) as fin:
+            # Use regex to find all PR numbers in the file and remove them
+            pr_re = re.compile(r"\#(\d+)")
+            prs_in_changelog = [
+                match.group(1) for match in pr_re.finditer(fin.read())
+            ]
+            to_remove = {pr for pr in prs if pr in prs_in_changelog}
+            for pr in to_remove:
+                print(f"Removing {pr} as already in CHANGELOG")
+                del prs[pr]
 
     # Print PRs with missing notes
     if prs:


Test patch:
diff --git a/numba/tests/support.py b/numba/tests/support.py
--- a/numba/tests/support.py
+++ b/numba/tests/support.py
@@ -150,6 +150,14 @@ def expected_failure_py312(fn):
                                           "parameter area' in "
                                           "LowerCall_64SVR4"))
 
+# LLVM PPC issue.
+# Sample error message:
+#   Invalid PPC CTR loop!
+#   UNREACHABLE executed at /llvm/lib/Target/PowerPC/PPCCTRLoops.cpp:179!
+skip_ppc64le_invalid_ctr_loop = unittest.skipIf(
+    platform.machine() == 'ppc64le',
+    "Invalid PPC CTR loop")
+
 # fenv.h on M1 may have various issues:
 # https://github.com/numba/numba/issues/7822#issuecomment-1065356758
 _uname = platform.uname()
diff --git a/numba/tests/test_parfors.py b/numba/tests/test_parfors.py
--- a/numba/tests/test_parfors.py
+++ b/numba/tests/test_parfors.py
@@ -44,7 +44,8 @@
                                  override_env_config, linux_only, tag,
                                  skip_parfors_unsupported, _32bit, needs_blas,
                                  needs_lapack, disabled_test, skip_unless_scipy,
-                                 needs_subprocess)
+                                 needs_subprocess,
+                                 skip_ppc64le_invalid_ctr_loop)
 from numba.core.extending import register_jitable
 from numba.core.bytecode import _fix_LOAD_GLOBAL_arg
 from numba.core import utils
@@ -2369,16 +2370,23 @@ def def_in_loop(x):
         self.assertEqual(expected, njit(parallel=True)(def_in_loop)(4))
 
     @needs_lapack  # use of np.linalg.solve
+    @skip_ppc64le_invalid_ctr_loop
     def test_issue9490_non_det_ssa_problem(self):
         cmd = [
             sys.executable,
             "-m",
             "numba.tests.parfor_iss9490_usecase",
         ]
-
+        envs = {
+            **os.environ,
+            # Reproducer consistently fail with the following hashseed.
+            "PYTHONHASHSEED": "1",
+            # See https://github.com/numba/numba/issues/9501
+            # for details of why num-thread pinning is needed.
+            "NUMBA_NUM_THREADS": "1",
+        }
         try:
-            subp.check_output(cmd, env={**os.environ,
-                                        "PYTHONHASHSEED": "1"},
+            subp.check_output(cmd, env=envs,
                               stderr=subp.STDOUT,
                               encoding='utf-8')
         except subp.CalledProcessError as e: