Repository: joblib/joblib
Pull request #1667 (2025-02-26 10:58:48)

MNT refactoring backend APIs
This is to refactor backend APIs to make them a bit more consistent and understandable.

Not finished yet.

codecov[bot]: ## [Codecov](https://app.codecov.io/gh/joblib/joblib/pull/1667?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=joblib) Report
Attention: Patch coverage is `26.66667%` with `11 lines` in your changes missing coverage. Please review.
> Project coverage is 70.03%. Comparing base [(`a34b3ad`)](https://app.codecov.io/gh/joblib/joblib/commit/a34b3ad6028929ff043a39647a7b9f1f9ca8498d?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=joblib) to head [(`3f96a5f`)](https://app.codecov.io/gh/joblib/joblib/commit/3f96a5fce78027c3a572f727d87edd8d0647f249?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=joblib).
> Report is 3 commits behind head on main.

| [Files with missing lines](https://app.codecov.io/gh/joblib/joblib/pull/1667?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=joblib) | Patch % | Lines |
|---|---|---|
| [joblib/\_parallel\_backends.py](https://app.codecov.io/gh/joblib/joblib/pull/1667?src=pr&el=tree&filepath=joblib%2F_parallel_backends.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=joblib#diff-am9ibGliL19wYXJhbGxlbF9iYWNrZW5kcy5weQ==) | 30.76% | [9 Missing :warning: ](https://app.codecov.io/gh/joblib/joblib/pull/1667?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=joblib) |
| [joblib/parallel.py](https://app.codecov.io/gh/joblib/joblib/pull/1667?src=pr&el=tree&filepath=joblib%2Fparallel.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=joblib#diff-am9ibGliL3BhcmFsbGVsLnB5) | 0.00% | [2 Missing :warning: ](https://app.codecov.io/gh/joblib/joblib/pull/1667?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=joblib) |

> :exclamation:  There is a different number of reports uploaded between BASE (a34b3ad) and HEAD (3f96a5f). Click for more details.
> 
> <details><summary>HEAD has 26 uploads less than BASE</summary>
>
>| Flag | BASE (a34b3ad) | HEAD (3f96a5f) |
>|------|------|------|
>||27|1|
></details>

<details><summary>Additional details and impacted files</summary>


```diff
@@             Coverage Diff             @@
##             main    #1667       +/-   ##
===========================================
- Coverage   95.73%   70.03%   -25.70%     
===========================================
  Files          46       46               
  Lines        7799     7833       +34     
===========================================
- Hits         7466     5486     -1980     
- Misses        333     2347     +2014     
```

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/joblib/joblib/pull/1667?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=joblib).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=joblib).

adrinjalali: Do we wanna merge this at this point?

Patch:
diff --git a/CHANGES.rst b/CHANGES.rst
index 77f45bf..497b6e7 100644
--- a/CHANGES.rst
+++ b/CHANGES.rst
@@ -4,8 +4,8 @@ Latest changes
 In development
 --------------
 
-- Drop support for Python 3.8.
-  https://github.com/joblib/joblib/pull/1669
+Memory:
+~~~~~~~
 
 - Enforce ``age_limit`` is a positive timedelta for ``Memory.reduce_size``,
   to avoid silently ignoring it.
@@ -20,16 +20,32 @@ In development
   if it is in cache memory.
   https://github.com/joblib/joblib/pull/1584
 
+- The Memory object now automatically creates a .gitignore file in its cache directory,
+  instructing git to ignore the entire folder.
+  https://github.com/joblib/joblib/pull/1674
+
+Parallel:
+~~~~~~~~~
+
 - Fixed a bug that caused the ``timeout`` parameter in ``joblib.Parallel`` to be
   ineffective when used along with ``return_as='generator_unordered'``.
   https://github.com/joblib/joblib/issues/1586
 
-- Support for Python 3.13 free-threaded has been added.
-  https://github.com/joblib/joblib/pull/1589
-
 - Pretty printing of ``Parallel`` execution progress when the number of tasks is
   known. https://github.com/joblib/joblib/pull/1608
 
+- Refactor and document the custom parallel backend API.
+  https://github.com/joblib/joblib/pull/1667
+
+Maintenance:
+~~~~~~~~~~~~
+
+- Drop support for Python 3.8.
+  https://github.com/joblib/joblib/pull/1669
+
+- Support for Python 3.13 free-threaded has been added.
+  https://github.com/joblib/joblib/pull/1589
+
 - Drop support for PyPy.
   https://github.com/joblib/joblib/pull/1670
 
@@ -40,9 +56,6 @@ In development
   the warnings that are emitted in this case.
   https://github.com/joblib/joblib/pull/1681
 
-- The Memory object now automatically creates a .gitignore file in its cache directory,
-  instructing git to ignore the entire folder.
-  https://github.com/joblib/joblib/pull/1674
 
 Release 1.4.2 -- 2024/05/02
 ---------------------------
diff --git a/doc/custom_parallel_backend.rst b/doc/custom_parallel_backend.rst
new file mode 100644
index 0000000..8562ca8
--- /dev/null
+++ b/doc/custom_parallel_backend.rst
@@ -0,0 +1,296 @@
+.. _custom_parallel_backend:
+
+==================================
+Parallel backend customization API
+==================================
+
+User can provide their own implementation of a parallel processing backend in
+addition to the ``'loky'``, ``'threading'``, ``'multiprocessing'`` backends
+provided by default.
+
+A new backend is defined by providing a backend factory, which is then
+registered with the :func:`joblib.register_parallel_backend` function by
+passing a name and the factory.
+
+For instance, to register a custom backend named ``'custom'`` with a factory ``MyCustomBackend``, you can call:
+
+.. code-block:: python
+
+    # Register the backend so it can be used with parallel_config
+    register_parallel_backend('custom', MyCustomBackend)
+
+This backend can then be used within the :func:`~joblib.parallel_config`
+context manager, as:
+
+.. code-block:: python
+
+    from joblib import Parallel, delayed, parallel_config
+
+    with parallel_config("custom"):
+        res = Parallel(2)(delayed(id)(i) for i in range(10))
+
+
+Minimal backend factory specification
+======================================
+
+The backend factory can be any callable that returns an instance of
+``ParallelBackendBase``. The complete specification of the backend factory is
+referenced in the `default backends source code`_.
+Here, we provide minimal details to implement a custom backend factory and
+the next section describes various possible customizations.
+
+The key methods for the backend factory are:
+
+- ``configure``: this method is called when creating a particular instance of
+  the ``Parallel`` object. It can allocate the resources needed for the
+  parallel computation and return the effective number of jobs that can be run
+  in parallel.
+- ``effective_n_jobs``: This method is called to determine the number of jobs
+  that can be run in parallel. It can be used to limit the number of jobs depending
+  on the specified ``n_jobs`` and the system it runs on.
+- ``terminate``: this method is called when the parallel computation is
+  finished. It can be used to clean up the resources allocated by the backend.
+- ``submit``: this method is called to launch the computation for a given task
+  in the ``Parallel`` call. The task is a single callable that takes no
+  argument and the method should return a future-like object that allows
+  tracking the task's progress. The ``callback`` argument passed to this method
+  is a callable that should be called when the task is completed, typically through
+  the ``add_done_callback`` method of the future-like object.
+- ``retrieve_result_callback``: This method is called within the callback
+  function passed to ``submit``. It is called with the same arguments provided
+  by the backend to the callback functionality -- typically the future-like object.
+  It should retrieve and return the result of the function executed in parallel.
+
+Below is a minimal example of a custom backend factory that uses a
+``ThreadPoolExecutor`` to run the tasks in parallel:
+
+.. code-block:: python
+
+    from concurrent.futures import ThreadPoolExecutor
+
+    from joblib import ParallelBackendBase
+    from joblib import register_parallel_backend
+
+
+    class MyCustomBackend(ParallelBackendBase):
+
+        supports_retrieve_callback = True
+
+        def configure(self, n_jobs=1, parallel=None, **backend_kwargs):
+            """Configure the backend for a specific instance of Parallel."""
+            self.n_jobs = n_jobs
+
+            n_jobs = self.effective_n_jobs(n_jobs)
+            self._executor = ThreadPoolExecutor(n_jobs)
+
+            # Return the effective number of jobs
+            return n_jobs
+
+        def terminate(self):
+            """Clean-up the resources associated with the backend."""
+            self._executor.shutdown()
+            self._executor = None
+
+        def effective_n_jobs(self, n_jobs):
+            """Determine the number of jobs that can be run in parallel."""
+            return n_jobs
+
+        def submit(self, func, callback):
+            """Schedule a function to be run and return a future-like object.
+
+            This method should return a future-like object that allow tracking
+            the progress of the task.
+
+            If ``supports_retrieve_callback`` is False, the return value of this
+            method is passed to ``retrieve_result`` instead of calling
+            ``retrieve_result_callback``.
+
+            Parameters
+            ----------
+            func: callable
+                The function to be run in parallel.
+
+            callback: callable
+                A callable that will be called when the task is completed. This callable
+                is a wrapper around ``retrieve_result_callback``. This should be added
+                to the future-like object returned by this method, so that the callback
+                is called when the task is completed.
+
+                For future-like backends, this can be achieved with something like
+                ``future.add_done_callback(callback)``.
+
+            Returns
+            -------
+            future: future-like
+                A future-like object to track the execution of the submitted function.
+            """
+            future = self._executor.submit(func)
+            future.add_done_callback(callback)
+            return future
+
+        def retrieve_result_callback(self, future):
+            """Called within the callback function passed to `submit`.
+
+            This method can customise how the result of the function is retrieved
+            from the future-like object.
+
+            Parameters
+            ----------
+            future: future-like
+                The future-like object returned by the `submit` method.
+
+            Returns
+            -------
+            result: object
+                The result of the function executed in parallel.
+            """
+            return future.result()
+
+
+Extra customizations
+--------------------
+
+The backend API offers several hooks that can be used to customize its behavior.
+
+Passing extra arguments to the backend
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+It is possible to register a backend class that has some mandatory
+constructor parameters such as the network address and connection credentials
+for a remote cluster computing service:
+
+.. code-block:: python
+
+    class MyCustomBackend(ParallelBackendBase):
+
+        def __init__(self, endpoint, api_key, nesting_level=0):
+           super().__init__(nesting_level=nesting_level)
+           self.endpoint = endpoint
+           self.api_key = api_key
+
+        ...
+        # Do something with self.endpoint and self.api_key somewhere in
+        # one of the method of the class
+
+    register_parallel_backend('custom', MyCustomBackend)
+
+The connection parameters can then be passed to the
+:func:`~joblib.parallel_config` context manager:
+
+.. code-block:: python
+
+    with parallel_config(backend='custom', endpoint='http://compute',
+                         api_key='42'):
+        Parallel()(delayed(some_function)(i) for i in range(10))
+
+Using the context manager can be helpful when using a third-party library that
+uses :class:`joblib.Parallel` internally while not exposing the ``backend``
+argument in its own API.
+
+Cancelling tasks
+~~~~~~~~~~~~~~~~
+
+If the backend allow to cancel tasks, the method ``abort_everything`` can be
+implemented to abort all the tasks that are currently running as soon as one of
+the tasks raises an exception. This can be useful to avoid wasting
+computational resources when the call will fail.
+
+This method have an extra parameters ``ensure_ready`` that informs the backend
+whether the error was part of a single call to ``Parallel`` or in a context
+manager block. In the case of a single call (``ensure_ready=False``), there is
+no need to re-spawn workers for future calls, while in the case of a context
+(``ensure_ready=True``), one could call ``configure`` to re-allocate computational
+resources.
+
+.. code-block:: python
+
+    def abort_everything(self, ensure_ready=True):
+        """Abort any running tasks
+
+        This is called when an exception has been raised when executing a task
+        and all the remaining tasks will be ignored and can therefore be
+        aborted to spare computation resources.
+
+        If ensure_ready is True, the backend should be left in an operating
+        state as future tasks might be re-submitted via that same backend
+        instance.
+
+        If ensure_ready is False, the implementer of this method can decide
+        to leave the backend in a closed / terminated state as no new task
+        are expected to be submitted to this backend.
+
+        Setting ensure_ready to False is an optimization that can be leveraged
+        when aborting tasks via killing processes from a local process pool
+        managed by the backend it-self: if we expect no new tasks, there is no
+        point in re-creating new workers.
+        """
+        pass
+
+Setting up Nested Parallelism
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The backend can also provide a method ``get_nested_backend`` that will be used
+to setup the default backend to be used in nested parallel calls.
+By default, the default backend is set to a thread-based backend for the first
+level and then falls back to a sequential backend to avoid spawning too many
+threads on the host.
+
+.. code-block:: python
+
+    def get_nested_backend(self):
+        """Backend instance to be used by nested Parallel calls.
+
+        By default a thread-based backend is used for the first level of
+        nesting. Beyond, switch to sequential backend to avoid spawning too
+        many threads on the host.
+        """
+        nesting_level = getattr(self, "nesting_level", 0) + 1
+        return LokyBackend(nesting_level=nesting_level), None
+
+Another nested parallelism that needs to be controlled is the numbers of thread
+in third-party C-level threadpools, *e.g.* OpenMP, MKL, or BLAS. In ``joblib``,
+this is controlled with the ``inner_max_num_threads`` argument that can be
+provided to the backend in the :func:`~joblib.parallel_config` context manager.
+To support this argument, the backend should set the
+``supports_inner_max_num_threads`` class attribute to ``True`` and accept the
+argument in the constructor to set this up in the workers. A helper to set this
+in the workers is to use environment variables provided by
+``self._prepare_worker_env(n_jobs)``.
+
+Third-party backend registration
+================================
+
+To be used seamlessly within :func:`~joblib.parallel_config`, external packages need
+to register their parallel backends so they are identified by joblib. This requires
+importing extra packages as in::
+
+   >>> import joblib
+   >>> with joblib.parallel_config(backend='custom'):  # doctest: +SKIP
+   ...     ...  # this fails
+   KeyError: 'custom'
+
+   # Import library to register external backend
+   >>> import my_custom_backend_library  # doctest: +SKIP
+   >>> with joblib.parallel_config(backend='custom'):  # doctest: +SKIP
+   ...     ... # this works
+
+This can be confusing for users.  To resolve this, external packages can
+safely register their backends directly within the joblib codebase by creating
+a small function that registers their backend, and including this function
+within the ``joblib.parallel.EXTERNAL_BACKENDS`` dictionary::
+
+   def _register_custom():
+       try:
+           import my_custom_library
+       except ImportError:
+           raise ImportError("an informative error message")
+
+   EXTERNAL_BACKENDS['custom'] = _register_custom
+
+This is subject to community review, but can reduce the confusion for users
+when relying on side effects of external package imports.
+
+
+.. Links to various page
+
+.. _`default backends source code`: https://github.com/joblib/joblib/blob/main/joblib/_parallel_backends.py
diff --git a/doc/index.rst b/doc/index.rst
index f7e99bc..ad11610 100644
--- a/doc/index.rst
+++ b/doc/index.rst
@@ -42,6 +42,7 @@ Introduction
     memory.rst
     parallel.rst
     persistence.rst
+    custom_parallel_backend.rst
     auto_examples/index
     developing.rst
 
diff --git a/doc/parallel.rst b/doc/parallel.rst
index b44e788..cf4a5ce 100644
--- a/doc/parallel.rst
+++ b/doc/parallel.rst
@@ -45,21 +45,24 @@ This generator enables reducing the memory footprint of
 aggregation, as illustrated in
 :ref:`sphx_glr_auto_examples_parallel_generator.py`.
 
-Future releases are planned to also support returning a generator that yields
-the results in the order of completion rather than the order of submission, by
-using ``return_as="generator_unordered"`` instead of ``return_as="generator"``.
-In this case the order of the outputs will depend on the concurrency of workers
-and will not be guaranteed to be deterministic, meaning the results can be
-yielded with a different order every time the code is executed.
+It is also possible to return a generator that yields the results in the order
+of completion rather than the order of submission, by using
+``return_as="generator_unordered"`` instead of ``return_as="generator"``.
+In this case the order of the outputs depends on the concurrency of workers
+and is not deterministic, meaning the results can be yielded with a different
+order every time the code is executed.
 
 Thread-based parallelism vs process-based parallelism
 =====================================================
 
 By default :class:`joblib.Parallel` uses the ``'loky'`` backend module to start
-separate Python worker processes to execute tasks concurrently on
-separate CPUs. This is a reasonable default for generic Python programs
-but can induce a significant overhead as the input and output data need
-to be serialized in a queue for communication with the worker processes (see
+separate Python worker processes to execute tasks concurrently on separate CPUs.
+But ``joblib`` also supports other backends to execute tasks concurrently, with
+different trade-offs (see :ref:`parallel_config_backend`).
+
+While ``'loky'`` is a reasonable default for generic Python programs , it
+can induce a significant overhead as the input and output data need to be
+serialized in a queue for communication with the worker processes (see
 :ref:`serialization_and_processes`).
 
 When you know that the function you are calling is based on a compiled
@@ -80,25 +83,12 @@ instead of the default ``"loky"`` backend:
     ...     delayed(sqrt)(i ** 2) for i in range(10))
     [0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]
 
-The :func:`~joblib.parallel_config` context manager helps selecting
-a specific backend implementation or setting the default number of jobs:
-
-    >>> from joblib import parallel_config
-    >>> with parallel_config(backend='threading', n_jobs=2):
-    ...    Parallel()(delayed(sqrt)(i ** 2) for i in range(10))
-    [0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]
-
-The latter is especially useful when calling a library that uses
-:class:`joblib.Parallel` internally without exposing backend selection as
-part of its public API.
-
 Note that the ``prefer="threads"`` option was introduced in joblib 0.12.
 In prior versions, the same effect could be achieved by hardcoding a
 specific backend implementation such as ``backend="threading"`` in the
 call to :class:`joblib.Parallel` but this is now considered a bad pattern
 (when done in a library) as it does not make it possible to override that
-choice with the :func:`~joblib.parallel_config` context manager.
-
+choice with the :func:`~joblib.parallel_config` context manager (see the next session).
 
 .. topic:: The loky backend may not always be available
 
@@ -106,6 +96,42 @@ choice with the :func:`~joblib.parallel_config` context manager.
    Pyodide). In this case the loky backend is not available and the
    default backend falls back to threading.
 
+.. _parallel_config_backend:
+
+Setting up ``joblib``'s backend with ``parallel_config``
+========================================================
+
+The :func:`~joblib.parallel_config` context manager allows selecting
+a specific backend implementation or setting the default parameters like the
+number of jobs in the backend as:
+
+    >>> from joblib import parallel_config
+    >>> with parallel_config(backend='threading', n_jobs=2):
+    ...    Parallel()(delayed(sqrt)(i ** 2) for i in range(10))
+    [0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]
+
+This context manager is especially useful when calling a library that uses
+:class:`joblib.Parallel` internally without exposing backend selection as
+part of its public API.
+
+The context manager allow to set various backend parameters:
+
+- ``n_jobs``: Set the number of jobs to run in parallel.
+- ``verbose``: Set the verbosity level of the backend.
+- ``prefer``: Set the preferred backend to use. Can be ``'processes'`` or
+  ``'threads'``.
+- ``require``: Set the requirement for the backend. Can be ``'sharedmem'``.
+- ``inner_max_num_threads``: Limit the maximum number of threads used by
+  third-party libraries that manage their own C-level thread-pool. This argument
+  is only supported by backend that set the ``supports_inner_max_num_threads``
+  class attribute to ``True``, such as the ``'loky'`` backend.
+- ``temp_folder, max_nbytes, mmap_mode``: Control the auto-memmapping behavior
+  of the backend. See :ref:`auto_memmapping_doc` for more details.
+
+Extra arguments passed in the :func:`~joblib.parallel_config` context are
+passed to the backend constructor, giving the possibility to set up the
+backend with additional parameters.
+
 In addition to the builtin joblib backends, there are several cluster-specific
 backends you can use:
 
@@ -115,6 +141,9 @@ backends you can use:
 * `Joblib Apache Spark Backend <https://github.com/joblib/joblib-spark>`_
   to distribute joblib tasks on a Spark cluster.
 
+It is also possible to provide a custom backend implementation, as described in
+:ref:`custom_parallel_backend`.
+
 .. _serialization_and_processes:
 
 Serialization & Processes
@@ -192,8 +221,13 @@ Note that the ``'loky'`` backend now used by default for process-based
 parallelism automatically tries to maintain and reuse a pool of workers
 by it-self even for calls without the context manager.
 
-.. include:: parallel_numpy.rst
 
+.. _auto_memmapping_doc:
+
+Working with numerical data in shared memory (memmapping)
+=========================================================
+
+.. include:: parallel_numpy.rst
 
 Avoiding over-subscription of CPU resources
 ============================================
@@ -245,81 +279,6 @@ In this example, 4 Python worker processes will be allowed to use 2 threads
 each, meaning that this program will be able to use up to 8 CPUs concurrently.
 
 
-Custom backend API
-==================
-
-.. versionadded:: 0.10
-
-User can provide their own implementation of a parallel processing
-backend in addition to the ``'loky'``, ``'threading'``,
-``'multiprocessing'`` backends provided by default. A backend is
-registered with the :func:`joblib.register_parallel_backend` function by
-passing a name and a backend factory.
-
-The backend factory can be any callable that returns an instance of
-``ParallelBackendBase``. Please refer to the `default backends source code`_ as
-a reference if you want to implement your own custom backend.
-
-.. _`default backends source code`: https://github.com/joblib/joblib/blob/main/joblib/_parallel_backends.py
-
-Note that it is possible to register a backend class that has some mandatory
-constructor parameters such as the network address and connection credentials
-for a remote cluster computing service::
-
-    class MyCustomBackend(ParallelBackendBase):
-
-        def __init__(self, endpoint, api_key):
-           self.endpoint = endpoint
-           self.api_key = api_key
-
-        ...
-        # Do something with self.endpoint and self.api_key somewhere in
-        # one of the method of the class
-
-    register_parallel_backend('custom', MyCustomBackend)
-
-The connection parameters can then be passed to the
-:func:`~joblib.parallel_config` context manager::
-
-    with parallel_config(backend='custom', endpoint='http://compute',
-                         api_key='42'):
-        Parallel()(delayed(some_function)(i) for i in range(10))
-
-Using the context manager can be helpful when using a third-party library that
-uses :class:`joblib.Parallel` internally while not exposing the ``backend``
-argument in its own API.
-
-
-A problem exists that external packages that register new parallel backends
-must now be imported explicitly for their backends to be identified by joblib::
-
-   >>> import joblib
-   >>> with joblib.parallel_config(backend='custom'):  # doctest: +SKIP
-   ...     ...  # this fails
-   KeyError: 'custom'
-
-   # Import library to register external backend
-   >>> import my_custom_backend_library  # doctest: +SKIP
-   >>> with joblib.parallel_config(backend='custom'):  # doctest: +SKIP
-   ...     ... # this works
-
-This can be confusing for users.  To resolve this, external packages can
-safely register their backends directly within the joblib codebase by creating
-a small function that registers their backend, and including this function
-within the ``joblib.parallel.EXTERNAL_PACKAGES`` dictionary::
-
-   def _register_custom():
-       try:
-           import my_custom_library
-       except ImportError:
-           raise ImportError("an informative error message")
-
-   EXTERNAL_BACKENDS['custom'] = _register_custom
-
-This is subject to community review, but can reduce the confusion for users
-when relying on side effects of external package imports.
-
-
 Old multiprocessing backend
 ===========================
 
diff --git a/doc/parallel_numpy.rst b/doc/parallel_numpy.rst
index 4c44990..1bfe04b 100644
--- a/doc/parallel_numpy.rst
+++ b/doc/parallel_numpy.rst
@@ -5,9 +5,6 @@
     >>> setup = getfixture('parallel_numpy_fixture')
     >>> fixture = setup(sys.modules[__name__])
 
-Working with numerical data in shared memory (memmapping)
-=========================================================
-
 By default the workers of the pool are real Python processes forked using the
 ``multiprocessing`` module of the Python standard library when ``n_jobs != 1``.
 The arguments passed as input to the ``Parallel`` call are serialized and
diff --git a/joblib/__init__.py b/joblib/__init__.py
index 93d2800..fa2c93c 100644
--- a/joblib/__init__.py
+++ b/joblib/__init__.py
@@ -112,6 +112,8 @@ __version__ = "1.5.dev0"
 import os
 
 from ._cloudpickle_wrapper import wrap_non_picklable_objects
+from ._parallel_backends import ParallelBackendBase
+from ._store_backends import StoreBackendBase
 from .compressor import register_compressor
 from .hashing import hash
 from .logger import Logger, PrintTime
@@ -128,24 +130,31 @@ from .parallel import (
 )
 
 __all__ = [
+    # On-disk result caching
     "Memory",
     "MemorizedResult",
-    "PrintTime",
-    "Logger",
-    "hash",
-    "dump",
-    "load",
+    "expires_after",
+    # Parallel code execution
     "Parallel",
     "delayed",
     "cpu_count",
     "effective_n_jobs",
-    "register_parallel_backend",
-    "parallel_backend",
-    "expires_after",
-    "register_store_backend",
-    "register_compressor",
     "wrap_non_picklable_objects",
+    # Context to change the backend globally
     "parallel_config",
+    "parallel_backend",
+    # Helpers to define and register store/parallel backends
+    "ParallelBackendBase",
+    "StoreBackendBase",
+    "register_compressor",
+    "register_parallel_backend",
+    "register_store_backend",
+    # Helpers kept for backward compatibility
+    "PrintTime",
+    "Logger",
+    "hash",
+    "dump",
+    "load",
 ]
 
 
diff --git a/joblib/_parallel_backends.py b/joblib/_parallel_backends.py
index ca55b8f..20ec44a 100644
--- a/joblib/_parallel_backends.py
+++ b/joblib/_parallel_backends.py
@@ -29,9 +29,14 @@ if mp is not None:
 class ParallelBackendBase(metaclass=ABCMeta):
     """Helper abc which defines all methods a ParallelBackend must implement"""
 
+    default_n_jobs = 1
+
     supports_inner_max_num_threads = False
+
+    # This flag was introduced for backward compatibility reasons.
+    # New backends should always set it to True and implement the
+    # `retrieve_result_callback` method.
     supports_retrieve_callback = False
-    default_n_jobs = 1
 
     @property
     def supports_return_generator(self):
@@ -78,20 +83,81 @@ class ParallelBackendBase(metaclass=ABCMeta):
         as long as all the workers have enough work to do.
         """
 
-    @abstractmethod
     def apply_async(self, func, callback=None):
-        """Schedule a func to be run"""
+        """Deprecated: implement `submit` instead."""
+        raise NotImplementedError("Implement `submit` instead.")
+
+    def submit(self, func, callback=None):
+        """Schedule a function to be run and return a future-like object.
+
+        This method should return a future-like object that allow tracking
+        the progress of the task.
+
+        If ``supports_retrieve_callback`` is False, the return value of this
+        method is passed to ``retrieve_result`` instead of calling
+        ``retrieve_result_callback``.
+
+        Parameters
+        ----------
+        func: callable
+            The function to be run in parallel.
+
+        callback: callable
+            A callable that will be called when the task is completed. This callable
+            is a wrapper around ``retrieve_result_callback``. This should be added
+            to the future-like object returned by this method, so that the callback
+            is called when the task is completed.
+
+            For future-like backends, this can be achieved with something like
+            ``future.add_done_callback(callback)``.
+
+        Returns
+        -------
+        future: future-like
+            A future-like object to track the execution of the submitted function.
+        """
+        warnings.warn(
+            "`apply_async` is deprecated, implement and use `submit` instead.",
+            DeprecationWarning,
+        )
+        return self.apply_async(func, callback)
 
     def retrieve_result_callback(self, out):
-        """Called within the callback function passed in apply_async.
+        """Called within the callback function passed to `submit`.
+
+        This method can customise how the result of the function is retrieved
+        from the future-like object.
+
+        Parameters
+        ----------
+        future: future-like
+            The future-like object returned by the `submit` method.
 
-        The argument of this function is the argument given to a callback in
-        the considered backend. It is supposed to return the outcome of a task
-        if it succeeded or raise the exception if it failed.
+        Returns
+        -------
+        result: object
+            The result of the function executed in parallel.
         """
 
+    def retrieve_result(self, out, timeout=None):
+        """Hook to retrieve the result when support_retrieve_callback=False.
+
+        The argument `out` is the result of the `submit` call. This method
+        should return the result of the computation or raise an exception if
+        the computation failed.
+        """
+        if self.supports_timeout:
+            return out.get(timeout=timeout)
+        else:
+            return out.get()
+
     def configure(
-        self, n_jobs=1, parallel=None, prefer=None, require=None, **backend_args
+        self,
+        n_jobs=1,
+        parallel=None,
+        prefer=None,
+        require=None,
+        **backend_args,
     ):
         """Reconfigure the backend and return the number of workers.
 
@@ -117,10 +183,6 @@ class ParallelBackendBase(metaclass=ABCMeta):
     def batch_completed(self, batch_size, duration):
         """Callback indicate how long it took to run a batch"""
 
-    def get_exceptions(self):
-        """List of exception types to be captured."""
-        return []
-
     def abort_everything(self, ensure_ready=True):
         """Abort any running tasks
 
@@ -158,23 +220,6 @@ class ParallelBackendBase(metaclass=ABCMeta):
         else:
             return ThreadingBackend(nesting_level=nesting_level), None
 
-    @contextlib.contextmanager
-    def retrieval_context(self):
-        """Context manager to manage an execution context.
-
-        Calls to Parallel.retrieve will be made inside this context.
-
-        By default, this does nothing. It may be useful for subclasses to
-        handle nested parallelism. In particular, it may be required to avoid
-        deadlocks if a backend manages a fixed number of workers, when those
-        workers may be asked to do nested Parallel calls. Without
-        'retrieval_context' this could lead to deadlock, as all the workers
-        managed by the backend may be "busy" waiting for the nested parallel
-        calls to finish, but the backend has no free workers to execute those
-        tasks.
-        """
-        yield
-
     def _prepare_worker_env(self, n_jobs):
         """Return environment variables limiting threadpools in external libs.
 
@@ -204,6 +249,23 @@ class ParallelBackendBase(metaclass=ABCMeta):
             env[self.TBB_ENABLE_IPC_VAR] = "1"
         return env
 
+    @contextlib.contextmanager
+    def retrieval_context(self):
+        """Context manager to manage an execution context.
+
+        Calls to Parallel.retrieve will be made inside this context.
+
+        By default, this does nothing. It may be useful for subclasses to
+        handle nested parallelism. In particular, it may be required to avoid
+        deadlocks if a backend manages a fixed number of workers, when those
+        workers may be asked to do nested Parallel calls. Without
+        'retrieval_context' this could lead to deadlock, as all the workers
+        managed by the backend may be "busy" waiting for the nested parallel
+        calls to finish, but the backend has no free workers to execute those
+        tasks.
+        """
+        yield
+
     @staticmethod
     def in_main_thread():
         return isinstance(threading.current_thread(), threading._MainThread)
@@ -227,7 +289,7 @@ class SequentialBackend(ParallelBackendBase):
             raise ValueError("n_jobs == 0 in Parallel has no meaning")
         return 1
 
-    def apply_async(self, func, callback=None):
+    def submit(self, func, callback=None):
         """Schedule a func to be run"""
         raise RuntimeError("Should never be called for SequentialBackend.")
 
@@ -268,10 +330,10 @@ class PoolManagerMixin(object):
             self._pool = None
 
     def _get_pool(self):
-        """Used by apply_async to make it possible to implement lazy init"""
+        """Used by `submit` to make it possible to implement lazy init"""
         return self._pool
 
-    def apply_async(self, func, callback=None):
+    def submit(self, func, callback=None):
         """Schedule a func to be run"""
         # Here, we need a wrapper to avoid crashes on KeyboardInterruptErrors.
         # We also call the callback on error, to make sure the pool does not
@@ -283,9 +345,13 @@ class PoolManagerMixin(object):
             error_callback=callback,
         )
 
-    def retrieve_result_callback(self, out):
+    def retrieve_result_callback(self, result):
         """Mimic concurrent.futures results, raising an error if needed."""
-        return _retrieve_traceback_capturing_wrapped_call(out)
+        # In the multiprocessing Pool API, the callback are called with the
+        # result value as an argument so `result`(`out`) is the output of
+        # job.get(). It's either the result or the exception raised while
+        # collecting the result.
+        return _retrieve_traceback_capturing_wrapped_call(result)
 
     def abort_everything(self, ensure_ready=True):
         """Shutdown the pool and restart a new one with the same parameters"""
@@ -599,16 +665,17 @@ class LokyBackend(AutoBatchingMixin, ParallelBackendBase):
             n_jobs = max(cpu_count() + 1 + n_jobs, 1)
         return n_jobs
 
-    def apply_async(self, func, callback=None):
+    def submit(self, func, callback=None):
         """Schedule a func to be run"""
         future = self._workers.submit(func)
         if callback is not None:
             future.add_done_callback(callback)
         return future
 
-    def retrieve_result_callback(self, out):
+    def retrieve_result_callback(self, future):
+        """Retrieve the result, here out is the future given by submit"""
         try:
-            return out.result()
+            return future.result()
         except ShutdownExecutorError:
             raise RuntimeError(
                 "The executor underlying Parallel has been shutdown. "
diff --git a/joblib/parallel.py b/joblib/parallel.py
index d97c692..cd4d16c 100644
--- a/joblib/parallel.py
+++ b/joblib/parallel.py
@@ -427,6 +427,12 @@ class parallel_config:
                     )
 
             backend = BACKENDS[backend](**backend_params)
+        else:
+            if len(backend_params) > 0:
+                raise ValueError(
+                    "Constructor parameters backend_params are only "
+                    "supported when backend is a string."
+                )
 
         if inner_max_num_threads is not None:
             msg = (
@@ -723,7 +729,7 @@ class BatchCompletionCallBack(object):
             self.status = TASK_PENDING
 
     def register_job(self, job):
-        """Register the object returned by `apply_async`."""
+        """Register the object returned by `submit`."""
         self.job = job
 
     def get_result(self, timeout):
@@ -754,10 +760,7 @@ class BatchCompletionCallBack(object):
 
         # For other backends, the main thread needs to run the retrieval step.
         try:
-            if backend.supports_timeout:
-                result = self.job.get(timeout=timeout)
-            else:
-                result = self.job.get()
+            result = backend.retrieve_result(self.job, timeout=timeout)
             outcome = dict(result=result, status=TASK_DONE)
         except BaseException as e:
             outcome = dict(result=e, status=TASK_ERROR)
@@ -797,7 +800,7 @@ class BatchCompletionCallBack(object):
     ##########################################################################
     #                     METHODS CALLED BY CALLBACK THREADS                 #
     ##########################################################################
-    def __call__(self, out):
+    def __call__(self, *args, **kwargs):
         """Function called by the callback thread after a job is completed."""
 
         # If the backend doesn't support callback retrievals, the next batch of
@@ -825,7 +828,7 @@ class BatchCompletionCallBack(object):
 
             # Retrieves the result of the task in the main process and dispatch
             # a new batch if needed.
-            job_succeeded = self._retrieve_result(out)
+            job_succeeded = self._retrieve_result(*args, **kwargs)
 
         if job_succeeded:
             self._dispatch_new()
@@ -1426,7 +1429,7 @@ class Parallel(Logger):
         # the queue by itself as soon as the callback is triggered to be able
         # to return the results in the order of completion.
 
-        job = self._backend.apply_async(batch, callback=batch_tracker)
+        job = self._backend.submit(batch, callback=batch_tracker)
         batch_tracker.register_job(job)
 
     def _register_new_job(self, batch_tracker):
@@ -1441,7 +1444,6 @@ class Parallel(Logger):
         This method is meant to be called concurrently by the multiprocessing
         callback. We rely on the thread-safety of dispatch_one_batch to protect
         against concurrent consumption of the unprotected iterator.
-
         """
         if not self.dispatch_one_batch(self._original_iterator):
             self._iterating = False


Test patch:
diff --git a/joblib/test/test_config.py b/joblib/test/test_config.py
index 992da05..f90544c 100644
--- a/joblib/test/test_config.py
+++ b/joblib/test/test_config.py
@@ -92,6 +92,10 @@ def test_parallel_config_constructor_params():
         with parallel_config(backend_param=1):
             pass
 
+    with raises(ValueError, match="only supported when backend is a string"):
+        with parallel_config(backend=BACKENDS[DEFAULT_BACKEND], backend_param=1):
+            pass
+
 
 def test_parallel_config_nested():
     # Check that nested configuration retrieves the info from the