Repository: numba/numba (language: python)

Issue:
Bad interaction (hang) involving TBB and subprocess fork and maybe MKL OMP
This is documenting an issue discovered during 0.59.1. By writing it in the issue tracker, hopefully it can be searchable in case similar problems occur in the future for us or others. The issue being described here is already fixed by https://github.com/numba/numba/pull/9480/commits/b39bdf2516174c6f723d6785bd5f70e7a301b4cd#. (Self reminder: cherrypick the fix to main and link this this issue)

The `test_issue9490_non_det_ssa_problem` testcase can hang on very specifically Python 3.9, NumPy 1.25 and Linux x86-64 at this line: https://github.com/numba/numba/blob/147c641fb337c41029e9e8832d7cb40be700e413/numba/tests/parfor_iss9490_usecase.py#L71
 

GDB reveals output:

```
(gdb) info threads
  Id   Target Id                                    Frame
* 1    Thread 0x7ffff7c39280 (LWP 3954038) "python" __pthread_clockjoin_ex (threadid=140736834631424, thread_return=0x0,
    clockid=<optimized out>, abstime=<optimized out>, block=<optimized out>) at pthread_join_common.c:145
  2    Thread 0x7fffdc49c700 (LWP 3954042) "python" syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  3    Thread 0x7fffdc09b700 (LWP 3954043) "python" 0x00007ffff7d3f89b in sched_yield () at ../sysdeps/unix/syscall-template.S:78
  4    Thread 0x7fffdbc9a700 (LWP 3954044) "python" 0x00007ffff7d3f89b in sched_yield () at ../sysdeps/unix/syscall-template.S:78
  5    Thread 0x7fffdb899700 (LWP 3954045) "python" 0x00007ffff7d3f89b in sched_yield () at ../sysdeps/unix/syscall-template.S:78
  6    Thread 0x7fffdb498700 (LWP 3954047) "python" syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  7    Thread 0x7fffdac96700 (LWP 3954048) "python" 0x00007ffff7d3f89b in sched_yield () at ../sysdeps/unix/syscall-template.S:78
  8    Thread 0x7fffdb097700 (LWP 3954046) "python" 0x00007ffff7d3f89b in sched_yield () at ../sysdeps/unix/syscall-template.S:78
  12   Thread 0x7fffd9c92700 (LWP 3954054) "python" 0x00007ffff7d3f89b in sched_yield () at ../sysdeps/unix/syscall-template.S:78
  13   Thread 0x7fffd9490700 (LWP 3954053) "python" 0x00007ffff7d3f89b in sched_yield () at ../sysdeps/unix/syscall-template.S:78
  14   Thread 0x7fffda895700 (LWP 3954050) "python" 0x00007ffff7d3f89b in sched_yield () at ../sysdeps/unix/syscall-template.S:78
  15   Thread 0x7fffd908f700 (LWP 3954055) "python" 0x00007ffff5b4349a in __kmp_acquire_ticket_lock ()
   from /home/siu/miniconda3/envs/testenv_bad/lib/python3.9/site-packages/mkl/../../../libiomp5.so
```


```

Thread 1 "python" received signal SIGINT, Interrupt.
__pthread_clockjoin_ex (threadid=140736414611200, thread_return=0x0, clockid=<optimized out>, abstime=<optimized out>, block=<optimized out>) at pthread_join_common.c:145
145	pthread_join_common.c: No such file or directory.
(gdb) bt
#0  __pthread_clockjoin_ex (threadid=140736414611200, thread_return=0x0, clockid=<optimized out>, abstime=<optimized out>, block=<optimized out>)
    at pthread_join_common.c:145
#1  0x00007fffdcba294f in tbb::detail::r1::rml::internal::thread_monitor::join (
    handle=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>)
    at /usr/local/src/conda/tbb-2021.8.0/src/tbb/rml_thread_monitor.h:201
#2  tbb::detail::r1::rml::private_worker::release_handle (join=<optimized out>, handle=<optimized out>)
    at /usr/local/src/conda/tbb-2021.8.0/src/tbb/private_server.cpp:231
#3  tbb::detail::r1::rml::private_worker::start_shutdown (this=0x7fffdc7eb900) at /usr/local/src/conda/tbb-2021.8.0/src/tbb/private_server.cpp:256
#4  tbb::detail::r1::rml::private_worker::start_shutdown (this=0x7fffdc7eb900) at /usr/local/src/conda/tbb-2021.8.0/src/tbb/private_server.cpp:236
#5  tbb::detail::r1::rml::private_server::request_close_connection (this=0x7fffdcad3f80) at /usr/local/src/conda/tbb-2021.8.0/src/tbb/private_server.cpp:188
#6  0x00007fffdcb9c473 in tbb::detail::r1::market::release (this=0x7fffdcacf580, is_public=<optimized out>, blocking_terminate=<optimized out>)
    at /usr/local/src/conda/tbb-2021.8.0/src/tbb/market.cpp:214
#7  0x00007fffdcb9770a in tbb::detail::r1::finalize_impl (handle=...,
    handle@entry=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>)
    at /usr/local/src/conda/tbb-2021.8.0/src/tbb/governor.cpp:290
#8  tbb::detail::r1::finalize (mode=1, mode@entry=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, handle=...,
    handle@entry=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>)
    at /usr/local/src/conda/tbb-2021.8.0/src/tbb/governor.cpp:303
#9  tbb::detail::r1::finalize (handle=..., mode=1) at /usr/local/src/conda/tbb-2021.8.0/src/tbb/governor.cpp:298
#10 0x00007fffdcb7d1db in prepare_fork() ()
   from /home/siu/miniconda3/envs/testenv_bad/lib/python3.9/site-packages/numba/np/ufunc/tbbpool.cpython-39-x86_64-linux-gnu.so
#11 0x00007ffff7cd1c50 in __run_fork_handlers (who=<optimized out>, who@entry=atfork_run_prepare, do_locking=do_locking@entry=true) at register-atfork.c:123
#12 0x00007ffff7d20094 in __libc_fork () at ../sysdeps/nptl/fork.c:58
#13 0x00007ffff7fad47b in subprocess_fork_exec (self=<optimized out>, args=<optimized out>) at /usr/local/src/conda/python-3.9.18/Modules/_posixsubprocess.c:844
#14 0x00000000005073b0 in cfunction_call (func=0x7ffff6bce5e0, args=<optimized out>, kwargs=<optimized out>)
    at /usr/local/src/conda/python-3.9.18/Objects/methodobject.c:552
```


Important observations here:
1. Thread 1 is stuck at `__pthread_clockjoin_ex` as part of a `prepare_fork` call for the TBB backend. TBB is known to have problems during fork. This explains the hang. But it is surprising for `numba.testing.assert_allclose` would cause a fork.
2. Thread 14 is from OMP in MKL. It is not confirmed whether having OMP threadpool have an effect in this hang.

Using PDB to trace the hanging line, it is discovered that NumPy 1.25+ introduced this lines which call subprocess thus fork:

https://github.com/numpy/numpy/blob/maintenance/1.25.x/numpy/testing/_private/utils.py#L1239-L1253

Together with the lazy import by `numba.testing.assert_allclose`, it explains the use of fork. 

As to why the test is not failing with NumPy 1.26:

- first, NumPy 1.26 conda package does not support Python 3.9. Thus we only test Python 3.10+. 
- Python 3.10 has several changes to `subprocess` which may have averted bugs with fork; e.g. https://github.com/python/cpython/issues/91401

The [fix] (https://github.com/numba/numba/blob/147c641fb337c41029e9e8832d7cb40be700e413/numba/tests/parfor_iss9490_usecase.py#L71) is done by forcing one Numba thread thus one TBB thread. That perhaps avoided `subprocess` fork bug. I didn't do further analysis since the bug is fixed. 






Discussion / hints:

Fix patch:
diff --git a/docs/source/release/0.59.1-notes.rst b/docs/source/release/0.59.1-notes.rst
new file mode 100644
index 00000000000..cc69758c036
--- /dev/null
+++ b/docs/source/release/0.59.1-notes.rst
@@ -0,0 +1,71 @@
+Version 0.59.1 (18 March 2024)
+------------------------------
+
+This is a bug-fix release to fix regressions in 0.59.0.
+
+
+CUDA API Changes
+~~~~~~~~~~~~~~~~
+
+Fixed caching of kernels that use target-specific overloads
+===========================================================
+
+Caching of kernels using target-specific overloads now works. This includes use
+of cooperative group sync, which is now implemented with a target-specific overload.
+
+(`PR-#9447 <https://github.com/numba/numba/pull/9447>`__)
+
+
+Performance Improvements and Changes
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Improvement to ``np.searchsorted``
+==================================
+
+Fixed a performance regression introduced in Numba 0.59 which made
+``np.searchsorted`` considerably slower.
+
+(`PR-#9448 <https://github.com/numba/numba/pull/9448>`__)
+
+
+Bug Fixes
+~~~~~~~~~
+
+Fix issues with ``np.searchsorted`` not handling ``np.datetime64``
+==================================================================
+
+This patch fixes two issues with ``np.searchsorted``. First, a regression is
+fixed in the support of ``np.datetime64``. Second, adopt ``NAT``-aware
+comparisons to fix mishandling of ``NAT`` value.
+
+(`PR-#9445 <https://github.com/numba/numba/pull/9445>`__)
+
+Allow use of Python 3.12 PEP-695 type parameter syntax
+======================================================
+
+A patch is added to properly parse the PEP 695 syntax. While Numba
+does not yet take advantage of type parameters, it will no longer erroneously
+reject functions defined with the new Python 3.12 syntax.
+
+(`PR-#9459 <https://github.com/numba/numba/pull/9459>`__)
+
+
+Pull-Requests:
+~~~~~~~~~~~~~~
+
+* PR `#9445 <https://github.com/numba/numba/pull/9445>`_: Fix #9427 np.searchsorted on datetime64 (`sklam <https://github.com/sklam>`_)
+* PR `#9447 <https://github.com/numba/numba/pull/9447>`_: Fix Issue #9432, caching of kernels using target-specific overloads (such as CG sync) (`gmarkall <https://github.com/gmarkall>`_ `sklam <https://github.com/sklam>`_)
+* PR `#9448 <https://github.com/numba/numba/pull/9448>`_: Fix `np.searchsorted` regression (`guilhermeleobas <https://github.com/guilhermeleobas>`_)
+* PR `#9449 <https://github.com/numba/numba/pull/9449>`_: Remove deprecated CondaEnvironment@1 (`sklam <https://github.com/sklam>`_)
+* PR `#9450 <https://github.com/numba/numba/pull/9450>`_: Fix gpuci versions (`gmarkall <https://github.com/gmarkall>`_)
+* PR `#9459 <https://github.com/numba/numba/pull/9459>`_: Support pep695 type param syntax (`sklam <https://github.com/sklam>`_)
+* PR `#9491 <https://github.com/numba/numba/pull/9491>`_: Fix non-deterministic bug caused by unstableness in SSA (`sklam <https://github.com/sklam>`_, `loicdtx <https://github.com/loicdtx>`_)
+
+
+Authors:
+~~~~~~~~
+
+* `gmarkall <https://github.com/gmarkall>`_
+* `guilhermeleobas <https://github.com/guilhermeleobas>`_
+* `sklam <https://github.com/sklam>`_
+* `loicdtx <https://github.com/loicdtx>`_
diff --git a/docs/source/user/installing.rst b/docs/source/user/installing.rst
index a0f06cfcb71..d2e8b8f7a9f 100644
--- a/docs/source/user/installing.rst
+++ b/docs/source/user/installing.rst
@@ -262,6 +262,8 @@ information.
 +----------++--------------+---------------------------+----------------------------+------------------------------+-------------------+-----------------------------+
 | Numba     | Release date | Python                    | NumPy                      | llvmlite                     | LLVM              | TBB                         |
 +===========+==============+===========================+============================+==============================+===================+=============================+
+| 0.59.1    | 2024-03-18   | 3.9.x <= version < 3.13   | 1.22 <= version < 1.27     | 0.42.x                       | 14.x              | 2021.6 <= version           |
++-----------+--------------+---------------------------+----------------------------+------------------------------+-------------------+-----------------------------+
 | 0.59.0    | 2024-01-31   | 3.9.x <= version < 3.13   | 1.22 <= version < 1.27     | 0.42.x                       | 14.x              | 2021.6 <= version           |
 +-----------+--------------+---------------------------+----------------------------+------------------------------+-------------------+-----------------------------+
 | 0.58.1    | 2023-10-17   | 3.8.x <= version < 3.12   | 1.22 <= version < 1.27     | 0.41.x                       | 14.x              | 2021.6 <= version           |
diff --git a/docs/upcoming_changes/9048.bug_fix.rst b/docs/upcoming_changes/9048.bug_fix.rst
deleted file mode 100644
index 21a68832f89..00000000000
--- a/docs/upcoming_changes/9048.bug_fix.rst
+++ /dev/null
@@ -1,17 +0,0 @@
-Dynamically Allocate Parfor Schedules
--------------------------------------
-
-This PR fixes an issue where a parallel region is executed in a loop
-many times.  The previous code used an alloca to allocate the parfor
-schedule on the stack but if there are many such parfors in a loop
-then the stack will overflow.  The new code does a pair of
-allocation/deallocation calls into the Numba parallel runtime before
-and after the parallel region respectively.  At the moment, these
-calls redirect to malloc/free although other mechanisms such as
-pooling are possible and may be implemented later.  This PR also
-adds a warning in cases where a prange loop is not converted to a
-parfor.  This can happen if there is exceptional control flow in the
-loop.  These are related in that the original issue had a prange
-loop that wasn't converted to a parfor and therefore all the parfors
-inside the body of the prange were running in parallel and adding to
-the stack each time.
diff --git a/docs/upcoming_changes/9049.bug_fix.rst b/docs/upcoming_changes/9049.bug_fix.rst
deleted file mode 100644
index d558b3e7389..00000000000
--- a/docs/upcoming_changes/9049.bug_fix.rst
+++ /dev/null
@@ -1,5 +0,0 @@
-Support multiple outputs in a ``@guvectorize`` function
-=======================================================
-
-This PR fixes `Numba #9058 <https://github.com/numba/numba/pull/9058>`_
-where it is now possible to call a guvectorize with multiple outputs.
diff --git a/docs/upcoming_changes/9089.bug_fix.rst b/docs/upcoming_changes/9089.bug_fix.rst
deleted file mode 100644
index a5755f9ed76..00000000000
--- a/docs/upcoming_changes/9089.bug_fix.rst
+++ /dev/null
@@ -1,4 +0,0 @@
-Handling of `None` args fixed in `PythonAPI.call`.
---------------------------------------------------
-
-Fixing segfault when `args=None` was passed to `PythonAPI.call`.
diff --git a/docs/upcoming_changes/9123.new_feature.rst b/docs/upcoming_changes/9123.new_feature.rst
deleted file mode 100644
index e4763978d70..00000000000
--- a/docs/upcoming_changes/9123.new_feature.rst
+++ /dev/null
@@ -1,4 +0,0 @@
-Add support for ufunc attributes and reduce
--------------------------------------------
-
-Support for ``ufunc.reduce`` and most ufunc attributes is added.
diff --git a/docs/upcoming_changes/9126.np_support.rst b/docs/upcoming_changes/9126.np_support.rst
deleted file mode 100644
index 5b6ac0a4560..00000000000
--- a/docs/upcoming_changes/9126.np_support.rst
+++ /dev/null
@@ -1,4 +0,0 @@
-Added support for np.indices() function.
-----------------------------------------
-
-Support is added for ``numpy.indices()``.
diff --git a/docs/upcoming_changes/9140.np_support.rst b/docs/upcoming_changes/9140.np_support.rst
deleted file mode 100644
index 3b6c6845df1..00000000000
--- a/docs/upcoming_changes/9140.np_support.rst
+++ /dev/null
@@ -1,4 +0,0 @@
-Added support for `np.polynomial.polynomial.Polynomial` class.
---------------------------------------------------------------
-
-Support is added for the `Polynomial` class from the package `np.polynomial.polynomial`.
\ No newline at end of file
diff --git a/docs/upcoming_changes/9141.np_support.rst b/docs/upcoming_changes/9141.np_support.rst
deleted file mode 100644
index 4d84fff47e3..00000000000
--- a/docs/upcoming_changes/9141.np_support.rst
+++ /dev/null
@@ -1,4 +0,0 @@
-Added support for functions ``np.polynomial.polyutils.as_series()``, as well as functions ``polydiv()``, ``polyint()``, ``polyval()`` from ``np.polynomial.polynomial``.
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-
-Support is added for ``np.polynomial.polyutils.as_series()``, ``np.polynomial.polynomial.polydiv()``, ``np.polynomial.polynomial.polyint()`` (only the first 2 arguments), ``np.polynomial.polynomial.polyval()`` (only the first 2 arguments).
diff --git a/docs/upcoming_changes/9144.bug_fix.rst b/docs/upcoming_changes/9144.bug_fix.rst
deleted file mode 100644
index 62c504c2d91..00000000000
--- a/docs/upcoming_changes/9144.bug_fix.rst
+++ /dev/null
@@ -1,5 +0,0 @@
-Fix propagation of literal values in PHI nodes.
------------------------------------------------
-
-Fixed a bug in the literal propagation pass where a PHI node could be wrongly
-replaced by a constant.
diff --git a/docs/upcoming_changes/9154.np_support.rst b/docs/upcoming_changes/9154.np_support.rst
deleted file mode 100644
index f59409dc0e3..00000000000
--- a/docs/upcoming_changes/9154.np_support.rst
+++ /dev/null
@@ -1,4 +0,0 @@
-Added support for np.unwrap() function.
----------------------------------------
-
-Support is added for ``numpy.unwrap()``. The ``axis`` argument is only supported when its value equals -1.
\ No newline at end of file
diff --git a/docs/upcoming_changes/9169.bug_fix.rst b/docs/upcoming_changes/9169.bug_fix.rst
deleted file mode 100644
index c2cd3f08c1c..00000000000
--- a/docs/upcoming_changes/9169.bug_fix.rst
+++ /dev/null
@@ -1,6 +0,0 @@
-``numpy.digitize`` implementation behaviour aligned with numpy
---------------------------------------------------------------
-
-The implementation of ``numpy.digitize`` is updated to behave per
-numpy in a wider set of cases, including where the supplied bins
-are not in fact monotonic.
diff --git a/docs/upcoming_changes/9189.bug_fix.rst b/docs/upcoming_changes/9189.bug_fix.rst
deleted file mode 100644
index 4d28f99b742..00000000000
--- a/docs/upcoming_changes/9189.bug_fix.rst
+++ /dev/null
@@ -1,16 +0,0 @@
-``numpy.searchsorted`` and ``numpy.sort`` behaviour updates
------------------------------------------------------------
-
-* ``numpy.searchsorted`` implementation updated to produce
-  identical outputs to numpy for a wider set of use cases,
-  including where the provided array `a` is in fact not
-  properly sorted.
-
-* ``numpy.searchsorted`` implementation bugfix for the case where
-  side='right' and the provided array `a` contains NaN(s).
-
-* ``numpy.searchsorted`` implementation extended to support complex
-  inputs.
-
-* ``numpy.sort`` (and ``array.sort``) implementation extended to
-  support sorting of complex data.
\ No newline at end of file
diff --git a/docs/upcoming_changes/9223.cuda.rst b/docs/upcoming_changes/9223.cuda.rst
deleted file mode 100644
index 7b95459fc64..00000000000
--- a/docs/upcoming_changes/9223.cuda.rst
+++ /dev/null
@@ -1,6 +0,0 @@
-Added support for compiling device functions with a C ABI
-=========================================================
-
-Support for compiling device functions with a C ABI through the
-:func:`compile_ptx() <numba.cuda.compile_ptx>` API, for easier interoperability
-with CUDA C/C++ and other languages.
diff --git a/docs/upcoming_changes/9235.cuda.rst b/docs/upcoming_changes/9235.cuda.rst
deleted file mode 100644
index 3c1704167c5..00000000000
--- a/docs/upcoming_changes/9235.cuda.rst
+++ /dev/null
@@ -1,5 +0,0 @@
-Make grid() and gridsize() use 64-bit integers
-----------------------------------------------
-
-`cuda.grid()` and `cuda.gridsize()` now use 64-bit integers, so they no longer
-overflow when the grid contains more than ``2 ** 31`` threads.
diff --git a/docs/upcoming_changes/9242.bug_fix.rst b/docs/upcoming_changes/9242.bug_fix.rst
deleted file mode 100644
index 1062af2155a..00000000000
--- a/docs/upcoming_changes/9242.bug_fix.rst
+++ /dev/null
@@ -1,6 +0,0 @@
-Fix SSA to consider variables where use is not dominated by the definition
-==========================================================================
-
-A SSA problem is fixed such that a conditionally defined variable will receive
-a phi node showing that there is a path where the variable is undefined.
-This affects extension code that relies on SSA behavior.
diff --git a/docs/upcoming_changes/9244.bug_fix.rst b/docs/upcoming_changes/9244.bug_fix.rst
deleted file mode 100644
index 09ad1614069..00000000000
--- a/docs/upcoming_changes/9244.bug_fix.rst
+++ /dev/null
@@ -1,26 +0,0 @@
-Fixed ``RecursionError`` in ``prange``
---------------------------------------
-
-A problem with certain loop patterns using ``prange`` leading to 
-``RecursionError`` in the compiler is fixed. An example of such loop is shown 
-below. The problem would cause the compiler to fall into an infinite recursive
-cycle trying to determine the definition of ``var1`` and ``var2``. The pattern
-involves definitions of variables within an if-else tree and not all branches 
-are defining the variables.
-
-.. code-block::
-
-    for i in prange(N):
-        for j in inner:
-            if cond1:
-                var1 = ...
-            elif cond2:
-                var1, var2 = ...
-
-            elif cond3:
-                pass
-
-            if cond4:
-                use(var1)
-                use(var2)
-
diff --git a/docs/upcoming_changes/9245.performance.rst b/docs/upcoming_changes/9245.performance.rst
deleted file mode 100644
index 142a6681e33..00000000000
--- a/docs/upcoming_changes/9245.performance.rst
+++ /dev/null
@@ -1,5 +0,0 @@
-Improvement to IR copying speed
--------------------------------
-
-Improvements were made to the deepcopying of ``FunctionIR``. 
-In one case, the ``InlineInlineables`` pass is 3x faster.
diff --git a/docs/upcoming_changes/9246.highlight.rst b/docs/upcoming_changes/9246.highlight.rst
deleted file mode 100644
index d72c843f03c..00000000000
--- a/docs/upcoming_changes/9246.highlight.rst
+++ /dev/null
@@ -1,13 +0,0 @@
-Python 3.12 Support
--------------------
-
-The standout feature of this release is the official support for Python 3.12 
-in Numba.
-
-Please note that profiling support is temporarily disabled in 
-this release (for Python 3.12) and several known issues have been identified 
-during development. The Numba team is actively working on resolving them. 
-Please refer to the respective issue pages 
-(`Numba #9289 <https://github.com/numba/numba/pull/9289>`_ and 
-`Numba #9291 <https://github.com/numba/numba/pull/9291>`_) 
-for a list of ongoing issues and updates on progress.
diff --git a/docs/upcoming_changes/9249.np_support.rst b/docs/upcoming_changes/9249.np_support.rst
deleted file mode 100644
index 09b3ad87b02..00000000000
--- a/docs/upcoming_changes/9249.np_support.rst
+++ /dev/null
@@ -1,4 +0,0 @@
-Adds support for checking if dtypes are equal.
-----------------------------------------------
-
-Support is added for checking if two dtype objects are equal, for example `assert X.dtype == np.dtype(np.float64)`.
diff --git a/docs/upcoming_changes/9267.cuda.rst b/docs/upcoming_changes/9267.cuda.rst
deleted file mode 100644
index a38366d6e9d..00000000000
--- a/docs/upcoming_changes/9267.cuda.rst
+++ /dev/null
@@ -1,5 +0,0 @@
-Prevent kernels being dropped by implementing the used list
-===========================================================
-
-Kernels are no longer dropped when being compiled and linked using nvJitLink,
-because they are added to the ``@"llvm.used"`` list.
diff --git a/docs/upcoming_changes/9279.cuda.rst b/docs/upcoming_changes/9279.cuda.rst
deleted file mode 100644
index da7ec034018..00000000000
--- a/docs/upcoming_changes/9279.cuda.rst
+++ /dev/null
@@ -1,5 +0,0 @@
-Support for Windows CUDA 12.0 toolkit conda packages
-====================================================
-
-The library paths used in CUDA toolkit 12.0 conda packages on Windows are
-added to the search paths used when detecting CUDA libraries.
\ No newline at end of file
diff --git a/docs/upcoming_changes/9296.bug_fix.rst b/docs/upcoming_changes/9296.bug_fix.rst
deleted file mode 100644
index 94bdc49f06b..00000000000
--- a/docs/upcoming_changes/9296.bug_fix.rst
+++ /dev/null
@@ -1,4 +0,0 @@
-Support negative axis in ufunc.reduce
-"""""""""""""""""""""""""""""""""""""
-
-Fixed a bug in ufunc.reduce to correctly handle negative axis values.
diff --git a/docs/upcoming_changes/9309.change.rst b/docs/upcoming_changes/9309.change.rst
deleted file mode 100644
index 7bf10a4f74d..00000000000
--- a/docs/upcoming_changes/9309.change.rst
+++ /dev/null
@@ -1,8 +0,0 @@
-Make test listing not invoke CPU compilation.
----------------------------------------------
-
-Numba's test listing command ``python -m numba.runtests -l`` has historically
-triggered CPU target compilation due to the way in which certain test functions
-were declared within the test suite. It has now been made such that the CPU
-target compiler is not invoked on test listing and a test is added to ensure
-that it remains the case.
diff --git a/docs/upcoming_changes/9310.highlight.rst b/docs/upcoming_changes/9310.highlight.rst
deleted file mode 100644
index 831b3d5b081..00000000000
--- a/docs/upcoming_changes/9310.highlight.rst
+++ /dev/null
@@ -1,5 +0,0 @@
-Move minimum supported Python version to 3.9.
----------------------------------------------
-
-Support for Python 3.8 has been removed, Numba's minimum supported Python
-version is now Python 3.9.
diff --git a/docs/upcoming_changes/9315.change.rst b/docs/upcoming_changes/9315.change.rst
deleted file mode 100644
index 39ef54e9ffc..00000000000
--- a/docs/upcoming_changes/9315.change.rst
+++ /dev/null
@@ -1,36 +0,0 @@
-Semantic differences due to Python 3.12 variable shadowing in comprehensions
-----------------------------------------------------------------------------
-
-Python 3.12 introduced a new bytecode ``LOAD_FAST_AND_CLEAR`` that is only used 
-in comprehensions. It has dynamic semantics that Numba cannot model. 
-
-For example,
-
-.. code-block:: python
-
-    def foo():
-        if False:
-            x = 1
-        [x for x in (1,)]
-        return x  # This return uses undefined variable
-
-The variable `x` is undefined at the return statement. Instead of raising an 
-``UnboundLocalError``, Numba will raise a ``TypingError`` at compile time if an 
-undefined variable is used.
-
-However, Numba cannot always detect undefined variables. 
-
-For example,
-
-.. code-block:: python
-
-    def foo(a):
-        [x for x in (0,)]
-        if a:
-            x = 3 + a
-        x += 10
-        return x
-
-Calling ``foo(0)`` returns ``10`` instead of raising ``UnboundLocalError``. 
-This is because Numba does not track variable liveness at runtime. 
-The return value is ``0 + 10`` since Numba zero-initializes undefined variables.
diff --git a/docs/upcoming_changes/9330.change.rst b/docs/upcoming_changes/9330.change.rst
deleted file mode 100644
index 3fcbb015391..00000000000
--- a/docs/upcoming_changes/9330.change.rst
+++ /dev/null
@@ -1,24 +0,0 @@
-Refactor and remove legacy APIs/testing internals.
-""""""""""""""""""""""""""""""""""""""""""""""""""
-
-A number of internally used functions have been removed to aid with general
-maintenance by reducing the number of ways in which it is possible to invoke
-compilation, specifically:
-
-* ``numba.core.compiler.compile_isolated`` is removed.
-* ``numba.tests.support.TestCase::run_nullary_func`` is removed.
-* ``numba.tests.support.CompilationCache`` is removed.
-
-Additionally, the concept of "nested context" is removed from
-``numba.core.registry.CPUTarget`` along with the implementation details.
-Maintainers of target extensions (those using the
-API in ``numba.core.target_extension`` to extend Numba support to
-custom/synthetic hardware) should note that the same can be deleted from
-target extension implementations of ``numba.core.descriptor.TargetDescriptor``
-if it is present. i.e. the ``nested_context`` method and associated
-implementation details can just be removed from the custom target's
-``TargetDescriptor``.
-
-Further, a bug was discovered, during the refactoring, in the typing of record
-arrays. It materialised that two record types that only differed in their
-mutability could alias, this has now been fixed.
diff --git a/docs/upcoming_changes/9334.bug_fix.rst b/docs/upcoming_changes/9334.bug_fix.rst
deleted file mode 100644
index 75dbb37ff4c..00000000000
--- a/docs/upcoming_changes/9334.bug_fix.rst
+++ /dev/null
@@ -1,11 +0,0 @@
-Fix issue with parfor reductions and Python 3.12.
-=================================================
-
-The parfor reduction code has certain expectations on the order of statements
-that it discovers, these are based on the code that previous versions of Numba
-generated.  With Python 3.12, one assignment that used to follow the
-reduction operator statement, such as a binop, is now moved to its own basic
-block.  This change reorders the set of discovered reduction nodes so that
-this assignment is right after the reduction operator as it was in previous
-Numba versions.  This only affects internal parfor reduction code and
-doesn't actually change the Numba IR.
diff --git a/docs/upcoming_changes/9335.infrastructure.rst b/docs/upcoming_changes/9335.infrastructure.rst
deleted file mode 100644
index f821374c4f8..00000000000
--- a/docs/upcoming_changes/9335.infrastructure.rst
+++ /dev/null
@@ -1,6 +0,0 @@
-Add validation capability for user generated towncrier ``.rst`` files.
-----------------------------------------------------------------------
-
-Added a validation script for user generated towncrier ``.rst`` files.
-The script will run as a part of towncrier Github workflow automatically
-on every PR.
diff --git a/docs/upcoming_changes/9336.improvement.rst b/docs/upcoming_changes/9336.improvement.rst
deleted file mode 100644
index 8e516577997..00000000000
--- a/docs/upcoming_changes/9336.improvement.rst
+++ /dev/null
@@ -1,11 +0,0 @@
-Add ``TargetLibraryInfo`` pass to CPU LLVM pipeline.
-""""""""""""""""""""""""""""""""""""""""""""""""""""
-
-The ``TargetLibraryInfo`` pass makes sure that the optimisations that take place
-during call simplification are appropriate for the target, without this the
-target is assumed to be Linux and code will be optimised to produce e.g. math
-symbols that do not exit on Windows. Historically this issue has been avoided
-through the use of Numba internal libraries carrying wrapped symbols, but doing
-so potentially detriments performance. As a result of this change Numba internal
-libraries are smaller and there is an increase in optimisation opportunity in
-code using ``exp2`` and ``log2`` functions.
diff --git a/docs/upcoming_changes/9341.new_feature.rst b/docs/upcoming_changes/9341.new_feature.rst
deleted file mode 100644
index 21649967a50..00000000000
--- a/docs/upcoming_changes/9341.new_feature.rst
+++ /dev/null
@@ -1,5 +0,0 @@
-Add a config variable to enable / disable the llvmlite memory manager
-=====================================================================
-
-A config variable to force enable or disable the llvmlite memory manager is
-added.
diff --git a/docs/upcoming_changes/9346.deprecation.rst b/docs/upcoming_changes/9346.deprecation.rst
deleted file mode 100644
index a45f297b847..00000000000
--- a/docs/upcoming_changes/9346.deprecation.rst
+++ /dev/null
@@ -1,8 +0,0 @@
-Explicitly setting ``NUMBA_CAPTURED_ERRORS=old_style`` will raise deprecation warnings
-======================================================================================
-
-As per deprecation schedule of old-style error-capturing, explicitly setting 
-``NUMBA_CAPTURED_ERRORS=old_style`` will raise deprecation warnings. 
-This release is the last to use "old_style" as the default.
-Details are documented at 
-https://numba.readthedocs.io/en/0.58.1/reference/deprecation.html#deprecation-of-old-style-numba-captured-errors
diff --git a/docs/upcoming_changes/9347.improvement.rst b/docs/upcoming_changes/9347.improvement.rst
deleted file mode 100644
index 0a5df436061..00000000000
--- a/docs/upcoming_changes/9347.improvement.rst
+++ /dev/null
@@ -1,9 +0,0 @@
-Numba deprecation warning classes are now subclasses of builtin ones
-====================================================================
-
-To help users manage and suppress deprecation warnings from Numba, the
-``NumbaDeprecationWarning`` and ``NumbaPendingDeprecationWarning`` classes are
-now subclasses of the builtin ``DeprecationWarning`` and
-``PendingDeprecationWarning`` respectively. Therefore, warning filters on
-``DeprecationWarning`` and ``PendingDeprecationWarning`` will apply to Numba
-deprecation warnings.
diff --git a/docs/upcoming_changes/9352.expired.rst b/docs/upcoming_changes/9352.expired.rst
deleted file mode 100644
index 80c51ecbc4e..00000000000
--- a/docs/upcoming_changes/9352.expired.rst
+++ /dev/null
@@ -1,8 +0,0 @@
-Object mode *fall-back* support has been removed.
-=================================================
-
-As per the deprecation schedule for Numba 0.59.0, support for
-"object mode fall-back" is removed from all Numba ``jit``-family decorators.
-Further, the default for the ``nopython`` key-word argument has been changed to
-``True``, this means that all Numba ``jit``-family decorated functions will now
-compile in ``nopython`` mode by default.
diff --git a/docs/upcoming_changes/9353.expired.rst b/docs/upcoming_changes/9353.expired.rst
deleted file mode 100644
index ac8c2659ccc..00000000000
--- a/docs/upcoming_changes/9353.expired.rst
+++ /dev/null
@@ -1,6 +0,0 @@
-Removal of deprecated API ``@numba.generated_jit``.
-===================================================
-
-As per the deprecation schedule for 0.59.0, support for
-``@numba.generated_jit`` has been removed. Use of ``@numba.extending.overload``
-and the high-level extension API is recommended as a replacement.
diff --git a/docs/upcoming_changes/9445.bug_fix.rst b/docs/upcoming_changes/9445.bug_fix.rst
deleted file mode 100644
index 0b9c4f4b06b..00000000000
--- a/docs/upcoming_changes/9445.bug_fix.rst
+++ /dev/null
@@ -1,6 +0,0 @@
-Fix issues with ``np.searchsorted`` not handling ``np.datetime64``
-------------------------------------------------------------------
-
-This patch fixes two issues with ``np.searchsorted``. First, a regression is 
-fixed in the support of ``np.datetime64``. Second, adopt ``NAT``-aware 
-comparisons to fix mishandling of ``NAT`` value.
diff --git a/docs/upcoming_changes/9447.cuda.rst b/docs/upcoming_changes/9447.cuda.rst
deleted file mode 100644
index d3c6285a996..00000000000
--- a/docs/upcoming_changes/9447.cuda.rst
+++ /dev/null
@@ -1,5 +0,0 @@
-Fixed caching of kernels that use target-specific overloads
------------------------------------------------------------
-
-Caching of kernels using target-specific overloads now works. This includes use
-of cooperative group sync, which is now implemented with a target-specific overload.
diff --git a/docs/upcoming_changes/9448.performance.rst b/docs/upcoming_changes/9448.performance.rst
deleted file mode 100644
index 830acfb4369..00000000000
--- a/docs/upcoming_changes/9448.performance.rst
+++ /dev/null
@@ -1,5 +0,0 @@
-Improvement to ``np.searchsorted``
-----------------------------------
-
-Fixed a performance regression introduced in Numba 0.59 which made
-``np.searchsorted`` considerably slower.
\ No newline at end of file
diff --git a/docs/upcoming_changes/9459.bug_fix.rst b/docs/upcoming_changes/9459.bug_fix.rst
deleted file mode 100644
index 721d705b659..00000000000
--- a/docs/upcoming_changes/9459.bug_fix.rst
+++ /dev/null
@@ -1,6 +0,0 @@
-Allow use of Python 3.12 PEP-695 type parameter syntax
-------------------------------------------------------
-
-A patch is added to properly parse the PEP 695 syntax. While Numba 
-does not yet take advantage of type parameters, it will no longer erroneously 
-reject functions defined with the new Python 3.12 syntax. 
diff --git a/maint/towncrier_checker.py b/maint/towncrier_checker.py
index e3511d37661..b05d1401726 100644
--- a/maint/towncrier_checker.py
+++ b/maint/towncrier_checker.py
@@ -66,16 +66,21 @@
 
     # Find PRs that are mentioned in CHANGE_LOG already
     # (most likely cherrypicked to previous bugfix releases)
-    with open(Path(os.path.dirname(__file__)) / ".." / "CHANGE_LOG") as fin:
-        # Use regex to find all PR numbers in the file and remove them
-        pr_re = re.compile(r"\#(\d+)")
-        prs_in_changelog = [
-            match.group(1) for match in pr_re.finditer(fin.read())
-        ]
-        to_remove = {pr for pr in prs if pr in prs_in_changelog}
-        for pr in to_remove:
-            print(f"Removing {pr} as already in CHANGELOG")
-            del prs[pr]
+    basepath = Path(os.path.dirname(__file__)) / ".."
+    oldchangelog = basepath / "CHANGE_LOG"
+    newchangelogs = (basepath / "docs" / "source" / "release").glob("*.rst")
+    for path in [oldchangelog, *newchangelogs]:
+        print('Checking', path)
+        with open(path) as fin:
+            # Use regex to find all PR numbers in the file and remove them
+            pr_re = re.compile(r"\#(\d+)")
+            prs_in_changelog = [
+                match.group(1) for match in pr_re.finditer(fin.read())
+            ]
+            to_remove = {pr for pr in prs if pr in prs_in_changelog}
+            for pr in to_remove:
+                print(f"Removing {pr} as already in CHANGELOG")
+                del prs[pr]
 
     # Print PRs with missing notes
     if prs:


Test patch:
diff --git a/numba/tests/support.py b/numba/tests/support.py
index b834f3afa4a..4aa4071342e 100644
--- a/numba/tests/support.py
+++ b/numba/tests/support.py
@@ -150,6 +150,14 @@ def expected_failure_py312(fn):
                                           "parameter area' in "
                                           "LowerCall_64SVR4"))
 
+# LLVM PPC issue.
+# Sample error message:
+#   Invalid PPC CTR loop!
+#   UNREACHABLE executed at /llvm/lib/Target/PowerPC/PPCCTRLoops.cpp:179!
+skip_ppc64le_invalid_ctr_loop = unittest.skipIf(
+    platform.machine() == 'ppc64le',
+    "Invalid PPC CTR loop")
+
 # fenv.h on M1 may have various issues:
 # https://github.com/numba/numba/issues/7822#issuecomment-1065356758
 _uname = platform.uname()
diff --git a/numba/tests/test_parfors.py b/numba/tests/test_parfors.py
index 3619ac201c2..a308c9490a3 100644
--- a/numba/tests/test_parfors.py
+++ b/numba/tests/test_parfors.py
@@ -44,7 +44,8 @@
                                  override_env_config, linux_only, tag,
                                  skip_parfors_unsupported, _32bit, needs_blas,
                                  needs_lapack, disabled_test, skip_unless_scipy,
-                                 needs_subprocess)
+                                 needs_subprocess,
+                                 skip_ppc64le_invalid_ctr_loop)
 from numba.core.extending import register_jitable
 from numba.core.bytecode import _fix_LOAD_GLOBAL_arg
 from numba.core import utils
@@ -2369,16 +2370,23 @@ def def_in_loop(x):
         self.assertEqual(expected, njit(parallel=True)(def_in_loop)(4))
 
     @needs_lapack  # use of np.linalg.solve
+    @skip_ppc64le_invalid_ctr_loop
     def test_issue9490_non_det_ssa_problem(self):
         cmd = [
             sys.executable,
             "-m",
             "numba.tests.parfor_iss9490_usecase",
         ]
-
+        envs = {
+            **os.environ,
+            # Reproducer consistently fail with the following hashseed.
+            "PYTHONHASHSEED": "1",
+            # See https://github.com/numba/numba/issues/9501
+            # for details of why num-thread pinning is needed.
+            "NUMBA_NUM_THREADS": "1",
+        }
         try:
-            subp.check_output(cmd, env={**os.environ,
-                                        "PYTHONHASHSEED": "1"},
+            subp.check_output(cmd, env=envs,
                               stderr=subp.STDOUT,
                               encoding='utf-8')
         except subp.CalledProcessError as e: