Repository: numba/numba

Issue:
Global locks cause semaphore leaks.
#### Description

Importing `numba` causes semaphores to leak.

#### Steps/Code to Reproduce

```python
import multiprocessing as mp

import numba


def w():
    pass


def main():
    ps = [mp.Process(target=w) for _ in range(4)]
    [p.start() for p in ps]
    [p.join() for p in ps]


if __name__ == '__main__':
    p = mp.get_context('spawn').Process(target=main)
    p.start()
    p.join()
```

#### Actual Results

```bash
$ python3 blah.py
/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 1 leaked semaphores to clean up at shutdown
  len(cache))
/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 1 leaked semaphores to clean up at shutdown
  len(cache))
/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 1 leaked semaphores to clean up at shutdown
  len(cache))
/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 1 leaked semaphores to clean up at shutdown
  len(cache))
$
```

#### Expected Results

```bash
$ python3 blah.py
$
```




Discussion:
This issue is causing a number of downstream issues:
- https://github.com/pytorch/pytorch/issues/23117
- https://github.com/librosa/librosa/issues/924

Also, this issue came up with `tqdm`: https://github.com/tqdm/tqdm/issues/611
Thanks for the report, I can reproduce.

The reason for the lock here:
https://github.com/numba/numba/blob/37ab1420fa87d02b635991b32f0faf5c65e8fc4d/numba/npyufunc/parallel.py#L265
is that it's predominantly protecting `fork(2)` across the threading layer backend load sequence as some upstream libraries like GNU OpenMP are not fork safe. It's been like this in production for about a year now and this is the first issue we've had reported, so it's positive that the impact is low! I think what's happening is as follows:

1. `numba` is imported as part of the load sequence from `if __name__ == "__main__"`
2. `1.` sets a module level `RLock` in `numba.npyufunc.parallel` using the default "fork" context.
3. A process is started in the `__main__` scope with the "spawn" context, so `fork(2), exec*(2)`, the replacement image will target the function `main`.
4. The replacement image imports `numba` as part of the load sequence when starting the function `main` and sets the `RLock` as before, this is done under a "fork" context.
5. The `main` function running in the replacement image has a default context of "spawn" once the process is fully initialized (by virtue of context start method inheritance), so the grandchildren of the original process that are created in this function are created via the `spawn` context. 
6. The grandchildren processes will start and create `RLock`s during import, these `RLock`s also have the `spawn` context.
7. The semaphores used by the `RLock`s in 6. leak because they are now genuinely new semaphores and as the reference is module level they are not garbage collected.

Given the purpose of this lock was to deal with known fork safety issues in the threading backend initialisation sequences (and general nervousness about init sequences in 3rd party libraries), I think it would be ok to make this global lock purely lock when the multiprocess context is either set to "fork" or it is implemented as "fork" by virtue of defaults on certain Python/Platform combinations.


To the other comments:
> This issue is causing a number of downstream issues:
> 
>     * [pytorch/pytorch#23117](https://github.com/pytorch/pytorch/issues/23117)

Seems like there's more than one instance of this same problematic pattern involved here. Numba needs fixing but equally other things may need to be too.
 
>     * [librosa/librosa#924](https://github.com/librosa/librosa/issues/924)

This isn't `librosa`s bug so think that ticket being closed is correct.

> Also, this issue came up with `tqdm`: [tqdm/tqdm#611](https://github.com/tqdm/tqdm/issues/611)

Seems to be the same pattern.

--------------------------
As an aside, I wonder if it makes sense for the new process to inherit the start method from the context in which it was started. This:

```python
import multiprocessing as mp

def main():
    print("def main():", mp.get_context())

if __name__ == '__main__':
    print("in __main__:", mp.get_context())
    p = mp.get_context('spawn').Process(target=main)
    p.start()
    p.join()
```
currently reports
```
in __main__: <multiprocessing.context.ForkContext object at 0x7f13c6598eb8>
def main(): <multiprocessing.context.SpawnContext object at 0x7f620f10ffd0>
```
but I'd have hoped a spawned process would use defaults for the platform, i.e. fork on linux.

Resolving patch:
diff --git a/numba/npyufunc/parallel.py b/numba/npyufunc/parallel.py
--- a/numba/npyufunc/parallel.py
+++ b/numba/npyufunc/parallel.py
@@ -13,9 +13,10 @@
 
 import os
 import platform
+import sys
 import warnings
 from threading import RLock as threadRLock
-from multiprocessing import RLock as procRLock
+import multiprocessing
 from contextlib import contextmanager
 
 import numpy as np
@@ -25,7 +26,7 @@
 
 from numba.npyufunc import ufuncbuilder
 from numba.numpy_support import as_dtype
-from numba import types, config
+from numba import types, config, utils
 from numba.npyufunc.wrappers import _wrapper_info
 
 
@@ -261,9 +262,30 @@ def build_gufunc_wrapper(py_func, cres, sin, sout, cache, is_parfors):
 
 
 _backend_init_thread_lock = threadRLock()
+
+_windows = sys.platform.startswith('win32')
+
+
+@contextmanager
+def _nop():
+    yield
+
+
 try:
-    _backend_init_process_lock = procRLock()
+    if utils.PY3:
+        # Force the use of an RLock in the case a fork was used to start the
+        # process and thereby the init sequence, some of the threading backend
+        # init sequences are not fork safe. Also, windows global mp locks seem
+        # to be fine.
+        if "fork" in multiprocessing.get_start_method() or _windows:
+            _backend_init_process_lock = multiprocessing.get_context().RLock()
+        else:
+            _backend_init_process_lock = _nop()
+    else:
+        # windows uses spawn so is fine, linux uses fork has the lock
+        _backend_init_process_lock = multiprocessing.RLock()
 except OSError as e:
+
     # probably lack of /dev/shm for semaphore writes, warn the user
     msg = ("Could not obtain multiprocessing lock due to OS level error: %s\n"
            "A likely cause of this problem is '/dev/shm' is missing or"
@@ -273,11 +295,7 @@ def build_gufunc_wrapper(py_func, cres, sin, sout, cache, is_parfors):
            "***\n")
     warnings.warn(msg % str(e))
 
-    @contextmanager
-    def nop():
-        yield
-
-    _backend_init_process_lock = nop()
+    _backend_init_process_lock = _nop()
 
 _is_initialized = False
 


Test patch:
diff --git a/numba/tests/orphaned_semaphore_usecase.py b/numba/tests/orphaned_semaphore_usecase.py
new file mode 100644
--- /dev/null
+++ b/numba/tests/orphaned_semaphore_usecase.py
@@ -0,0 +1,24 @@
+# See issue: https://github.com/numba/numba/issues/4348
+# this file must be run in isolation to replicate, test:
+# numba.tests.test_parallel_backend.TestInitSafetyIssues.test_orphaned_semaphore
+# does this to check semaphores are not leaking.
+
+import multiprocessing as mp
+
+import numba # noqa, deliberately unused, here to test import is safe
+
+
+def w():
+    pass
+
+
+def main():
+    ps = [mp.Process(target=w) for _ in range(4)]
+    [p.start() for p in ps]
+    [p.join() for p in ps]
+
+
+if __name__ == '__main__':
+    p = mp.get_context('spawn').Process(target=main)
+    p.start()
+    p.join()
diff --git a/numba/tests/test_parallel_backend.py b/numba/tests/test_parallel_backend.py
--- a/numba/tests/test_parallel_backend.py
+++ b/numba/tests/test_parallel_backend.py
@@ -811,5 +811,47 @@ def test_serial_parent_explicit_mp_fork_par_child_then_par_parent(self):
             print(out, err)
 
 
+@parfors_skip_unsupported
+class TestInitSafetyIssues(TestCase):
+
+    _DEBUG = False
+
+    @linux_only # only linux can leak semaphores
+    @skip_unless_py3 # need multiprocessing.get_context to obtain spawn on linux
+    def test_orphaned_semaphore(self):
+        # sys path injection and separate usecase module to make sure everything
+        # is importable by children of multiprocessing
+
+        def run_cmd(cmdline):
+            popen = subprocess.Popen(cmdline,
+                                     stdout=subprocess.PIPE,
+                                     stderr=subprocess.PIPE,)
+            # finish in 5 minutes or kill it
+            timeout = threading.Timer(5 * 60., popen.kill)
+            try:
+                timeout.start()
+                out, err = popen.communicate()
+                if popen.returncode != 0:
+                    raise AssertionError(
+                        "process failed with code %s: stderr follows\n%s\n" %
+                        (popen.returncode, err.decode()))
+            finally:
+                timeout.cancel()
+            return out.decode(), err.decode()
+
+        test_file = os.path.join(os.path.dirname(__file__),
+                                 "orphaned_semaphore_usecase.py")
+
+        cmdline = [sys.executable, test_file]
+        out, err = run_cmd(cmdline)
+
+        # assert no semaphore leaks reported on stderr
+        self.assertNotIn("leaked semaphore", err)
+
+        if self._DEBUG:
+            print("OUT:", out)
+            print("ERR:", err)
+
+
 if __name__ == '__main__':
     unittest.main()