Repository: joblib/joblib
Pull request #1379 (2023-02-01 08:12:20)

CI/TST provide more informative traceback in case of a deadlock in tests
A set of changes proposed by @tomMoral in #588 

codecov[bot]: # [Codecov](https://codecov.io/gh/joblib/joblib/pull/1379?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=joblib) Report
Base: **94.01**% // Head: **68.47**% // Decreases project coverage by **`-25.55%`** :warning:
> Coverage data is based on head [(`eca5f2a`)](https://codecov.io/gh/joblib/joblib/pull/1379?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=joblib) compared to base [(`6836640`)](https://codecov.io/gh/joblib/joblib/commit/6836640abba55611d6e57f20338ea54b3e27f296?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=joblib).
> Patch coverage: 66.66% of modified lines in pull request are covered.

<details><summary>Additional details and impacted files</summary>


```diff
@@             Coverage Diff             @@
##           master    #1379       +/-   ##
===========================================
- Coverage   94.01%   68.47%   -25.55%     
===========================================
  Files          52       52               
  Lines        7300     7121      -179     
===========================================
- Hits         6863     4876     -1987     
- Misses        437     2245     +1808     
```


| [Impacted Files](https://codecov.io/gh/joblib/joblib/pull/1379?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=joblib) | Coverage Δ | |
|---|---|---|
| [joblib/test/test\_memmapping.py](https://codecov.io/gh/joblib/joblib/pull/1379?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=joblib#diff-am9ibGliL3Rlc3QvdGVzdF9tZW1tYXBwaW5nLnB5) | `25.04% <ø> (-74.21%)` | :arrow_down: |
| [joblib/test/test\_numpy\_pickle.py](https://codecov.io/gh/joblib/joblib/pull/1379?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=joblib#diff-am9ibGliL3Rlc3QvdGVzdF9udW1weV9waWNrbGUucHk=) | `42.50% <0.00%> (-51.74%)` | :arrow_down: |
| [joblib/test/test\_testing.py](https://codecov.io/gh/joblib/joblib/pull/1379?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=joblib#diff-am9ibGliL3Rlc3QvdGVzdF90ZXN0aW5nLnB5) | `100.00% <ø> (ø)` | |
| [joblib/testing.py](https://codecov.io/gh/joblib/joblib/pull/1379?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=joblib#diff-am9ibGliL3Rlc3RpbmcucHk=) | `85.71% <80.00%> (-9.53%)` | :arrow_down: |
| [joblib/test/test\_dask.py](https://codecov.io/gh/joblib/joblib/pull/1379?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=joblib#diff-am9ibGliL3Rlc3QvdGVzdF9kYXNrLnB5) | `3.92% <0.00%> (-87.27%)` | :arrow_down: |
| [joblib/\_dask.py](https://codecov.io/gh/joblib/joblib/pull/1379?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=joblib#diff-am9ibGliL19kYXNrLnB5) | `24.01% <0.00%> (-68.67%)` | :arrow_down: |
| [joblib/test/testutils.py](https://codecov.io/gh/joblib/joblib/pull/1379?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=joblib#diff-am9ibGliL3Rlc3QvdGVzdHV0aWxzLnB5) | `33.33% <0.00%> (-66.67%)` | :arrow_down: |
| [joblib/test/test\_hashing.py](https://codecov.io/gh/joblib/joblib/pull/1379?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=joblib#diff-am9ibGliL3Rlc3QvdGVzdF9oYXNoaW5nLnB5) | `49.26% <0.00%> (-49.82%)` | :arrow_down: |
| ... and [31 more](https://codecov.io/gh/joblib/joblib/pull/1379?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=joblib) | |

Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=joblib). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=joblib)

</details>

[:umbrella: View full report at Codecov](https://codecov.io/gh/joblib/joblib/pull/1379?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=joblib).   
:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=joblib).


Patch:
diff --git a/CHANGES.rst b/CHANGES.rst
index 72e983e..3f1bf8e 100644
--- a/CHANGES.rst
+++ b/CHANGES.rst
@@ -22,6 +22,16 @@ In development
   in May 2022) that it should be safe to remove this.
   https://github.com/joblib/joblib/pull/1361
 
+- A warning is raised when a pickling error occurs during caching operations.
+  In version 1.5, this warning will be turned into an error. For all other
+  errors, a new warning has been introduced: `joblib.memory.CacheWarning`.
+  https://github.com/joblib/joblib/pull/1359
+
+- Avoid (module, name) collisions when caching nested functions. This fix
+  changes the module name of nested functions, invalidating caches from
+  previous versions of Joblib.
+  https://github.com/joblib/joblib/pull/1374
+
 Release 1.2.0
 -------------
 
diff --git a/azure-pipelines.yml b/azure-pipelines.yml
index 8837a7e..a9cdaa1 100644
--- a/azure-pipelines.yml
+++ b/azure-pipelines.yml
@@ -23,7 +23,7 @@ jobs:
       displayName: Add conda to PATH
     - bash: sudo chown -R $USER $CONDA
       displayName: Take ownership of conda installation
-    - bash: conda create --name flake8_env --yes flake8
+    - bash: conda create --name flake8_env --yes "flake8<6"
       displayName: Install flake8
     - bash: |
         if [[ $BUILD_SOURCEVERSIONMESSAGE =~ \[lint\ skip\] ]]; then
diff --git a/joblib/_store_backends.py b/joblib/_store_backends.py
index e96f306..21a4958 100644
--- a/joblib/_store_backends.py
+++ b/joblib/_store_backends.py
@@ -1,5 +1,6 @@
 """Storage providers backends for Memory caching."""
 
+from pickle import PicklingError
 import re
 import os
 import os.path
@@ -20,6 +21,11 @@ CacheItemInfo = collections.namedtuple('CacheItemInfo',
                                        'path size last_access')
 
 
+class CacheWarning(Warning):
+    """Warning to capture dump failures except for PicklingError."""
+    pass
+
+
 def concurrency_safe_write(object_to_write, filename, write_func):
     """Writes an object into a unique file in a concurrency-safe way."""
     thread_id = id(threading.current_thread())
@@ -185,12 +191,24 @@ class StoreBackendMixin(object):
 
             def write_func(to_write, dest_filename):
                 with self._open_item(dest_filename, "wb") as f:
-                    numpy_pickle.dump(to_write, f,
-                                      compress=self.compress)
+                    try:
+                        numpy_pickle.dump(to_write, f, compress=self.compress)
+                    except PicklingError as e:
+                        # TODO(1.5) turn into error
+                        warnings.warn(
+                            "Unable to cache to disk: failed to pickle "
+                            "output. In version 1.5 this will raise an "
+                            f"exception. Exception: {e}.",
+                            FutureWarning
+                        )
 
             self._concurrency_safe_write(item, filename, write_func)
-        except:  # noqa: E722
-            " Race condition in the creation of the directory "
+        except Exception as e:  # noqa: E722
+            warnings.warn(
+                "Unable to cache to disk. Possibly a race condition in the "
+                f"creation of the directory. Exception: {e}.",
+                CacheWarning
+            )
 
     def clear_item(self, path):
         """Clear the item at the path, given as a list of strings."""
diff --git a/joblib/func_inspect.py b/joblib/func_inspect.py
index d334a2b..58d9751 100644
--- a/joblib/func_inspect.py
+++ b/joblib/func_inspect.py
@@ -166,6 +166,10 @@ def get_func_name(func, resolv_alias=True, win_characters=True):
         if hasattr(func, 'func_globals') and name in func.func_globals:
             if not func.func_globals[name] is func:
                 name = '%s-alias' % name
+    if hasattr(func, '__qualname__') and func.__qualname__ != name:
+        # Extend the module name in case of nested functions to avoid
+        # (module, name) collisions
+        module.extend(func.__qualname__.split(".")[:-1])
     if inspect.ismethod(func):
         # We need to add the name of the class
         if hasattr(func, 'im_class'):
diff --git a/joblib/memory.py b/joblib/memory.py
index dc95d2b..b27c1fc 100644
--- a/joblib/memory.py
+++ b/joblib/memory.py
@@ -30,6 +30,7 @@ from .func_inspect import format_call
 from .func_inspect import format_signature
 from .logger import Logger, format_time, pformat
 from ._store_backends import StoreBackendBase, FileSystemStoreBackend
+from ._store_backends import CacheWarning  # noqa
 
 
 FIRST_LINE_TEXT = "# first line:"
diff --git a/pyproject.toml b/pyproject.toml
new file mode 100644
index 0000000..fd51f1d
--- /dev/null
+++ b/pyproject.toml
@@ -0,0 +1,65 @@
+[build-system]
+requires = ["setuptools>=61.2"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "joblib"
+authors = [{name = "Gael Varoquaux", email = "gael.varoquaux@normalesup.org"}]
+license = {text = "BSD 3-Clause"}
+description = "Lightweight pipelining with Python functions"
+classifiers = [
+    "Development Status :: 5 - Production/Stable",
+    "Environment :: Console",
+    "Intended Audience :: Developers",
+    "Intended Audience :: Science/Research",
+    "Intended Audience :: Education",
+    "License :: OSI Approved :: BSD License",
+    "Operating System :: OS Independent",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.7",
+    "Programming Language :: Python :: 3.8",
+    "Programming Language :: Python :: 3.9",
+    "Programming Language :: Python :: 3.10",
+    "Topic :: Scientific/Engineering",
+    "Topic :: Utilities",
+    "Topic :: Software Development :: Libraries",
+]
+requires-python = ">=3.7"
+dynamic = ["version"]
+
+[project.readme]
+file = "README.rst"
+content-type = "text/x-rst"
+
+[project.urls]
+Homepage = "https://joblib.readthedocs.io"
+Source = "https://github.com/joblib/joblib"
+
+[tool.setuptools]
+packages = [
+    "joblib",
+    "joblib.test",
+    "joblib.test.data",
+    "joblib.externals",
+    "joblib.externals.cloudpickle",
+    "joblib.externals.loky",
+    "joblib.externals.loky.backend",
+]
+platforms = ["any"]
+include-package-data = false
+
+[tool.setuptools.package-data]
+"joblib.test" = [
+    "data/*.gz",
+    "data/*.gzip",
+    "data/*.bz2",
+    "data/*.xz",
+    "data/*.lzma",
+    "data/*.pkl",
+    "data/*.npy",
+    "data/*.npy.z",
+]
+
+[tool.setuptools.dynamic]
+version = {attr = "joblib.__version__"}
+
diff --git a/setup.cfg b/setup.cfg
index e3dfb34..65601e0 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -20,6 +20,3 @@ testpaths = joblib
 # E402: module level import not at top of file
 ignore=E402,W504
 exclude=joblib/externals/*
-
-[metadata]
-license_file = LICENSE.txt
diff --git a/setup.py b/setup.py
index d9c642f..2553605 100755
--- a/setup.py
+++ b/setup.py
@@ -1,61 +1,6 @@
 #!/usr/bin/env python
 
 from setuptools import setup
-import pathlib
-import joblib
 
-# Get the long description from the README file
-here = pathlib.Path(__file__).parent.resolve()
-long_description = (here / 'README.rst').read_text(encoding='utf-8')
-
-
-setup(
-    name='joblib',
-    version=joblib.__version__,
-    author='Gael Varoquaux',
-    author_email='gael.varoquaux@normalesup.org',
-    url='https://joblib.readthedocs.io',
-    project_urls={
-        'Source': 'https://github.com/joblib/joblib',
-    },
-    license='BSD',
-    description="Lightweight pipelining with Python functions",
-    long_description=long_description,
-    long_description_content_type="text/x-rst",
-    classifiers=[
-        'Development Status :: 5 - Production/Stable',
-        'Environment :: Console',
-        'Intended Audience :: Developers',
-        'Intended Audience :: Science/Research',
-        'Intended Audience :: Education',
-        'License :: OSI Approved :: BSD License',
-        'Operating System :: OS Independent',
-        'Programming Language :: Python :: 3',
-        'Programming Language :: Python :: 3.7',
-        'Programming Language :: Python :: 3.8',
-        'Programming Language :: Python :: 3.9',
-        'Programming Language :: Python :: 3.10',
-        'Topic :: Scientific/Engineering',
-        'Topic :: Utilities',
-        'Topic :: Software Development :: Libraries',
-    ],
-    platforms='any',
-    package_data={
-        'joblib.test': [
-            'data/*.gz',
-            'data/*.gzip',
-            'data/*.bz2',
-            'data/*.xz',
-            'data/*.lzma',
-            'data/*.pkl',
-            'data/*.npy',
-            'data/*.npy.z',
-        ]
-    },
-    packages=[
-        'joblib', 'joblib.test', 'joblib.test.data',
-        'joblib.externals', 'joblib.externals.cloudpickle',
-        'joblib.externals.loky', 'joblib.externals.loky.backend',
-    ],
-    python_requires='>=3.7',
-)
+if __name__ == "__main__":
+    setup()


Test patch:
diff --git a/conftest.py b/conftest.py
index 5a7919f..53413de 100644
--- a/conftest.py
+++ b/conftest.py
@@ -1,9 +1,11 @@
 import os
 
+import logging
+import faulthandler
+
 import pytest
 from _pytest.doctest import DoctestItem
 
-import logging
 from joblib.parallel import mp
 from joblib.backports import LooseVersion
 try:
@@ -53,6 +55,11 @@ def pytest_configure(config):
         log.handlers[0].setFormatter(logging.Formatter(
             '[%(levelname)s:%(processName)s:%(threadName)s] %(message)s'))
 
+    # Some CI runs failed with hanging processes that were not terminated
+    # with the timeout. To make sure we always get a proper trace, set a large
+    # enough dump_traceback_later to kill the process with a report.
+    faulthandler.dump_traceback_later(30 * 60, exit=True)
+
     DEFAULT_BACKEND = os.environ.get(
         "JOBLIB_TESTS_DEFAULT_PARALLEL_BACKEND", None
     )
@@ -66,12 +73,14 @@ def pytest_configure(config):
 
 
 def pytest_unconfigure(config):
+
     # Setup a global traceback printer callback to debug deadlocks that
     # would happen once pytest has completed: for instance in atexit
     # finalizers. At this point the stdout/stderr capture of pytest
-    # should be disabled.
+    # should be disabled. Note that we cancel the global dump_traceback_later
+    # to waiting for too long.
+    faulthandler.cancel_dump_traceback_later()
 
     # Note that we also use a shorter timeout for the per-test callback
     # configured via the pytest-timeout extension.
-    import faulthandler
     faulthandler.dump_traceback_later(60, exit=True)
diff --git a/joblib/test/test_func_inspect.py b/joblib/test/test_func_inspect.py
index b3b7b7b..a2a010e 100644
--- a/joblib/test/test_func_inspect.py
+++ b/joblib/test/test_func_inspect.py
@@ -146,6 +146,26 @@ def test_func_name_on_inner_func(cached_func):
     assert get_func_name(cached_func)[1] == 'cached_func_inner'
 
 
+def test_func_name_collision_on_inner_func():
+    # Check that two functions defining and caching an inner function
+    # with the same do not cause (module, name) collision
+    def f():
+        def inner_func():
+            return  # pragma: no cover
+        return get_func_name(inner_func)
+
+    def g():
+        def inner_func():
+            return  # pragma: no cover
+        return get_func_name(inner_func)
+
+    module, name = f()
+    other_module, other_name = g()
+
+    assert name == other_name
+    assert module != other_module
+
+
 def test_func_inspect_errors():
     # Check that func_inspect is robust and will work on weird objects
     assert get_func_name('a'.lower)[-1] == 'lower'
diff --git a/joblib/test/test_memmapping.py b/joblib/test/test_memmapping.py
index 4d298d0..af90a96 100644
--- a/joblib/test/test_memmapping.py
+++ b/joblib/test/test_memmapping.py
@@ -798,9 +798,9 @@ def test_child_raises_parent_exits_cleanly(backend):
 
         def get_temp_folder(parallel_obj, backend):
             if "{b}" == "loky":
-                return Path(p._backend._workers._temp_folder)
+                return Path(parallel_obj._backend._workers._temp_folder)
             else:
-                return Path(p._backend._pool._temp_folder)
+                return Path(parallel_obj._backend._pool._temp_folder)
 
 
         if __name__ == "__main__":
diff --git a/joblib/test/test_numpy_pickle.py b/joblib/test/test_numpy_pickle.py
index c9d1d5b..9fee585 100644
--- a/joblib/test/test_numpy_pickle.py
+++ b/joblib/test/test_numpy_pickle.py
@@ -280,8 +280,9 @@ def test_compress_mmap_mode_warning(tmpdir):
     numpy_pickle.dump(a, this_filename, compress=1)
     with warns(UserWarning) as warninfo:
         numpy_pickle.load(this_filename, mmap_mode='r+')
+    debug_msg = "\n".join([str(w) for w in warninfo])
     warninfo = [w.message for w in warninfo]
-    assert len(warninfo) == 1
+    assert len(warninfo) == 1, debug_msg
     assert (
         str(warninfo[0]) ==
         'mmap_mode "r+" is not compatible with compressed '
@@ -932,27 +933,27 @@ def test_pickle_in_socket():
     listener.bind(_ADDR)
     listener.listen(1)
 
-    client = socket.create_connection(_ADDR)
-    server, client_addr = listener.accept()
+    with socket.create_connection(_ADDR) as client:
+        server, client_addr = listener.accept()
 
-    with server.makefile("wb") as sf:
-        numpy_pickle.dump(test_array, sf)
+        with server.makefile("wb") as sf:
+            numpy_pickle.dump(test_array, sf)
 
-    with client.makefile("rb") as cf:
-        array_reloaded = numpy_pickle.load(cf)
+        with client.makefile("rb") as cf:
+            array_reloaded = numpy_pickle.load(cf)
 
-    np.testing.assert_array_equal(array_reloaded, test_array)
+        np.testing.assert_array_equal(array_reloaded, test_array)
 
-    # Check that a byte-aligned numpy array written in a file can be send over
-    # a socket and then read on the other side
-    bytes_to_send = io.BytesIO()
-    numpy_pickle.dump(test_array, bytes_to_send)
-    server.send(bytes_to_send.getvalue())
+        # Check that a byte-aligned numpy array written in a file can be send
+        # over a socket and then read on the other side
+        bytes_to_send = io.BytesIO()
+        numpy_pickle.dump(test_array, bytes_to_send)
+        server.send(bytes_to_send.getvalue())
 
-    with client.makefile("rb") as cf:
-        array_reloaded = numpy_pickle.load(cf)
+        with client.makefile("rb") as cf:
+            array_reloaded = numpy_pickle.load(cf)
 
-    np.testing.assert_array_equal(array_reloaded, test_array)
+        np.testing.assert_array_equal(array_reloaded, test_array)
 
 
 @with_numpy
diff --git a/joblib/test/test_store_backends.py b/joblib/test/test_store_backends.py
index 2c6198f..e5db167 100644
--- a/joblib/test/test_store_backends.py
+++ b/joblib/test/test_store_backends.py
@@ -1,3 +1,4 @@
+
 try:
     # Python 2.7: use the C pickle to speed up
     # test_concurrency_safe_write which pickles big python objects
@@ -5,13 +6,20 @@ try:
 except ImportError:
     import pickle as cpickle
 import functools
+from pickle import PicklingError
 import time
 
+import pytest
+
 from joblib.testing import parametrize, timeout
 from joblib.test.common import with_multiprocessing
 from joblib.backports import concurrency_safe_rename
 from joblib import Parallel, delayed
-from joblib._store_backends import concurrency_safe_write
+from joblib._store_backends import (
+    concurrency_safe_write,
+    FileSystemStoreBackend,
+    CacheWarning,
+)
 
 
 def write_func(output, filename):
@@ -54,3 +62,33 @@ def test_concurrency_safe_write(tmpdir, backend):
              if i % 3 != 2 else load_func for i in range(12)]
     Parallel(n_jobs=2, backend=backend)(
         delayed(func)(obj, filename) for func in funcs)
+
+
+def test_warning_on_dump_failure(tmpdir):
+    # Check that a warning is raised when the dump fails for any reason but
+    # a PicklingError.
+    class UnpicklableObject(object):
+        def __reduce__(self):
+            raise RuntimeError("some exception")
+
+    backend = FileSystemStoreBackend()
+    backend.location = tmpdir.join('test_warning_on_pickling_error').strpath
+    backend.compress = None
+
+    with pytest.warns(CacheWarning, match="some exception"):
+        backend.dump_item("testpath", UnpicklableObject())
+
+
+def test_warning_on_pickling_error(tmpdir):
+    # This is separate from test_warning_on_dump_failure because in the
+    # future we will turn this into an exception.
+    class UnpicklableObject(object):
+        def __reduce__(self):
+            raise PicklingError("not picklable")
+
+    backend = FileSystemStoreBackend()
+    backend.location = tmpdir.join('test_warning_on_pickling_error').strpath
+    backend.compress = None
+
+    with pytest.warns(FutureWarning, match="not picklable"):
+        backend.dump_item("testpath", UnpicklableObject())
diff --git a/joblib/test/test_testing.py b/joblib/test/test_testing.py
index 39ac880..cc94463 100644
--- a/joblib/test/test_testing.py
+++ b/joblib/test/test_testing.py
@@ -58,7 +58,9 @@ def test_check_subprocess_call_timeout():
         'sys.stdout.flush()',
         'sys.stderr.write("before sleep on stderr")',
         'sys.stderr.flush()',
-        'time.sleep(1.1)',
+        # We need to sleep for at least 2 * timeout seconds in case the SIGKILL
+        # is triggered.
+        'time.sleep(2.1)',
         'print("process should have be killed before")',
         'sys.stdout.flush()'])
 
diff --git a/joblib/testing.py b/joblib/testing.py
index f8939f0..caab7d2 100644
--- a/joblib/testing.py
+++ b/joblib/testing.py
@@ -40,20 +40,39 @@ def check_subprocess_call(cmd, timeout=5, stdout_regex=None,
                           stderr_regex=None):
     """Runs a command in a subprocess with timeout in seconds.
 
+    A SIGTERM is sent after `timeout` and if it does not terminate, a
+    SIGKILL is sent after `2 * timeout`.
+
     Also checks returncode is zero, stdout if stdout_regex is set, and
     stderr if stderr_regex is set.
     """
     proc = subprocess.Popen(cmd, stdout=subprocess.PIPE,
                             stderr=subprocess.PIPE)
 
-    def kill_process():
-        warnings.warn("Timeout running {}".format(cmd))
+    def terminate_process():  # pragma: no cover
+        """
+        Attempt to terminate a leftover process spawned during test execution:
+        ideally this should not be needed but can help avoid clogging the CI
+        workers in case of deadlocks.
+        """
+        warnings.warn(f"Timeout running {cmd}")
+        proc.terminate()
+
+    def kill_process():  # pragma: no cover
+        """
+        Kill a leftover process spawned during test execution: ideally this
+        should not be needed but can help avoid clogging the CI workers in
+        case of deadlocks.
+        """
+        warnings.warn(f"Timeout running {cmd}")
         proc.kill()
 
     try:
         if timeout is not None:
-            timer = threading.Timer(timeout, kill_process)
-            timer.start()
+            terminate_timer = threading.Timer(timeout, terminate_process)
+            terminate_timer.start()
+            kill_timer = threading.Timer(2 * timeout, kill_process)
+            kill_timer.start()
         stdout, stderr = proc.communicate()
         stdout, stderr = stdout.decode(), stderr.decode()
         if proc.returncode != 0:
@@ -76,4 +95,5 @@ def check_subprocess_call(cmd, timeout=5, stdout_regex=None,
 
     finally:
         if timeout is not None:
-            timer.cancel()
+            terminate_timer.cancel()
+            kill_timer.cancel()