Repository: numba/numba

Issue:
Importing numba is slow
Hello,

on my Windows workstation with >1200 pip installed packages, `import numba` is very slow:
```
py -3.7-64 -X importtime -c "import numba"
import time: self [us] | cumulative | imported package
import time:   3069020 |    3235357 |           pkg_resources
import time:      2526 |    3237883 |         numba.entrypoints
import time:     31084 |    4448066 | numba
```
The culprit is https://github.com/numba/numba/blob/310fe841c4744129347c5ef8e68d193a4d26c392/numba/entrypoints.py#L3

Importing `pkg_resources` is known to be excruciatingly slow. See https://github.com/pypa/setuptools/issues/510

How about replacing the use of `pkg_resources` for entrypoints by [`importlib_metadata`](https://importlib-metadata.readthedocs.io/en/latest/using.html#entry-points) or [`entrypoints`](https://entrypoints.readthedocs.io/en/latest/) if possible?

<!--

Thanks for opening an issue! To help the Numba team handle your information
efficiently, please first ensure that there is no other issue present that
already describes the issue you have
(search at https://github.com/numba/numba/issues?&q=is%3Aissue).

For more general "how do I do X?" type questions, please speak to us in real
time on https://gitter.im/numba/numba or post to the Numba mailing list
https://groups.google.com/a/continuum.io/forum/#!forum/numba-users.

-->

## Reporting a bug

<!--

Before submitting a bug report please ensure that you can check off these boxes:

-->

- [x] I am using the latest released version of Numba (most recent is visible in
 the change log (https://github.com/numba/numba/blob/master/CHANGE_LOG).
- [x] I have included below a minimal working reproducer (if you are unsure how
 to write one see http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports).

<!--

Please include details of the bug here, including, if applicable, what you
expected to happen!

-->



Discussion:
Thanks for the report. IIRC this approach mirrors that taken in pandas for the plotting backends https://github.com/pandas-dev/pandas/blob/0c2b1db198ce628ca889ad25c1179fff4ab3337c/pandas/plotting/_core.py#L1613-L1615 but it would appear they delay this scanning until needed. It seems like the suggested fixes involve taking on dependencies. CC @seibert, thoughts?
Adding a dependency on `import_metadata` I think would be fine.  It is pure Python, and already packaged in Anaconda as well.  It also appears in the standard library as of Python 3.8:

https://docs.python.org/3/library/importlib.metadata.html

so this seems like heading toward where Python is already going.
Task here is to remove the use of `pkg_resources` and replace with `importlib_metadata` and fix up the builds for that dependency chain.

Resolving patch:
diff --git a/numba/core/entrypoints.py b/numba/core/entrypoints.py
--- a/numba/core/entrypoints.py
+++ b/numba/core/entrypoints.py
@@ -1,17 +1,30 @@
 import logging
 import warnings
 
-from pkg_resources import iter_entry_points
+from numba.core.config import PYVERSION
+
+if PYVERSION < (3, 9):
+    try:
+        import importlib_metadata
+    except ImportError as ex:
+        raise ImportError(
+            "importlib_metadata backport is required for Python version < 3.9, "
+            "try:\n"
+            "$ conda/pip install importlib_metadata"
+        ) from ex
+else:
+    from importlib import metadata as importlib_metadata
+
 
 _already_initialized = False
 logger = logging.getLogger(__name__)
 
 
 def init_all():
-    '''Execute all `numba_extensions` entry points with the name `init`
+    """Execute all `numba_extensions` entry points with the name `init`
 
     If extensions have already been initialized, this function does nothing.
-    '''
+    """
     global _already_initialized
     if _already_initialized:
         return
@@ -19,13 +32,27 @@ def init_all():
     # Must put this here to avoid extensions re-triggering initialization
     _already_initialized = True
 
-    for entry_point in iter_entry_points('numba_extensions', 'init'):
+    def load_ep(entry_point):
+        """Loads a given entry point. Warns and logs on failure.
+        """
         logger.debug('Loading extension: %s', entry_point)
         try:
             func = entry_point.load()
             func()
         except Exception as e:
-            msg = "Numba extension module '{}' failed to load due to '{}({})'."
-            warnings.warn(msg.format(entry_point.module_name, type(e).__name__,
-                                     str(e)), stacklevel=2)
+            msg = (f"Numba extension module '{entry_point.module}' "
+                   f"failed to load due to '{type(e).__name__}({str(e)})'.")
+            warnings.warn(msg, stacklevel=3)
             logger.debug('Extension loading failed for: %s', entry_point)
+
+    eps = importlib_metadata.entry_points()
+    # Split, Python 3.10+ and importlib_metadata 3.6+ have the "selectable"
+    # interface, versions prior to that do not. See "compatibility note" in:
+    # https://docs.python.org/3.10/library/importlib.metadata.html#entry-points
+    if hasattr(eps, 'select'):
+        for entry_point in eps.select(group="numba_extensions", name="init"):
+            load_ep(entry_point)
+    else:
+        for entry_point in eps.get("numba_extensions", ()):
+            if entry_point.name == "init":
+                load_ep(entry_point)
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -361,6 +361,7 @@ def check_file_at_path(path2file):
     'llvmlite >={},<{}'.format(min_llvmlite_version, max_llvmlite_version),
     'numpy >={}'.format(min_numpy_run_version),
     'setuptools',
+    'importlib_metadata; python_version < "3.9"',
 ]
 
 metadata = dict(


Test patch:
diff --git a/numba/tests/test_entrypoints.py b/numba/tests/test_entrypoints.py
--- a/numba/tests/test_entrypoints.py
+++ b/numba/tests/test_entrypoints.py
@@ -1,16 +1,22 @@
 import sys
+from unittest import mock
+
 import types
 import warnings
 import unittest
 import os
 import subprocess
 import threading
-import pkg_resources
 
-from numba import njit
+from numba import config, njit
 from numba.tests.support import TestCase
 from numba.testing.main import _TIMEOUT as _RUNNER_TIMEOUT
 
+if config.PYVERSION < (3, 9):
+    import importlib_metadata
+else:
+    from importlib import metadata as importlib_metadata
+
 _TEST_TIMEOUT = _RUNNER_TIMEOUT - 60.
 
 
@@ -31,45 +37,35 @@ def test_init_entrypoint(self):
         # loosely based on Pandas test from:
         #   https://github.com/pandas-dev/pandas/pull/27488
 
-        # FIXME: Python 2 workaround because nonlocal doesn't exist
-        counters = {'init': 0}
-
-        def init_function():
-            counters['init'] += 1
-
-        mod = types.ModuleType("_test_numba_extension")
-        mod.init_func = init_function
+        mod = mock.Mock(__name__='_test_numba_extension')
 
         try:
             # will remove this module at the end of the test
             sys.modules[mod.__name__] = mod
 
-            # We are registering an entry point using the "numba" package
-            # ("distribution" in pkg_resources-speak) itself, though these are
-            # normally registered by other packages.
-            dist = "numba"
-            entrypoints = pkg_resources.get_entry_map(dist)
-            my_entrypoint = pkg_resources.EntryPoint(
-                "init",  # name of entry point
-                mod.__name__,  # module with entry point object
-                attrs=['init_func'],  # name of entry point object
-                dist=pkg_resources.get_distribution(dist)
+            my_entrypoint = importlib_metadata.EntryPoint(
+                'init', '_test_numba_extension:init_func', 'numba_extensions',
             )
-            entrypoints.setdefault('numba_extensions',
-                                   {})['init'] = my_entrypoint
 
-            from numba.core import entrypoints
-            # Allow reinitialization
-            entrypoints._already_initialized = False
+            with mock.patch.object(
+                importlib_metadata,
+                'entry_points',
+                return_value={'numba_extensions': (my_entrypoint,)},
+            ):
+
+                from numba.core import entrypoints
 
-            entrypoints.init_all()
+                # Allow reinitialization
+                entrypoints._already_initialized = False
+
+                entrypoints.init_all()
 
-            # was our init function called?
-            self.assertEqual(counters['init'], 1)
+                # was our init function called?
+                mod.init_func.assert_called_once()
 
-            # ensure we do not initialize twice
-            entrypoints.init_all()
-            self.assertEqual(counters['init'], 1)
+                # ensure we do not initialize twice
+                entrypoints.init_all()
+                mod.init_func.assert_called_once()
         finally:
             # remove fake module
             if mod.__name__ in sys.modules:
@@ -79,50 +75,41 @@ def test_entrypoint_tolerance(self):
         # loosely based on Pandas test from:
         #   https://github.com/pandas-dev/pandas/pull/27488
 
-        # FIXME: Python 2 workaround because nonlocal doesn't exist
-        counters = {'init': 0}
-
-        def init_function():
-            counters['init'] += 1
-            raise ValueError("broken")
-
-        mod = types.ModuleType("_test_numba_bad_extension")
-        mod.init_func = init_function
+        mod = mock.Mock(__name__='_test_numba_bad_extension')
+        mod.configure_mock(**{'init_func.side_effect': ValueError('broken')})
 
         try:
             # will remove this module at the end of the test
             sys.modules[mod.__name__] = mod
 
-            # We are registering an entry point using the "numba" package
-            # ("distribution" in pkg_resources-speak) itself, though these are
-            # normally registered by other packages.
-            dist = "numba"
-            entrypoints = pkg_resources.get_entry_map(dist)
-            my_entrypoint = pkg_resources.EntryPoint(
-                "init",  # name of entry point
-                mod.__name__,  # module with entry point object
-                attrs=['init_func'],  # name of entry point object
-                dist=pkg_resources.get_distribution(dist)
+            my_entrypoint = importlib_metadata.EntryPoint(
+                'init',
+                '_test_numba_bad_extension:init_func',
+                'numba_extensions',
             )
-            entrypoints.setdefault('numba_extensions',
-                                   {})['init'] = my_entrypoint
 
-            from numba.core import entrypoints
-            # Allow reinitialization
-            entrypoints._already_initialized = False
+            with mock.patch.object(
+                importlib_metadata,
+                'entry_points',
+                return_value={'numba_extensions': (my_entrypoint,)},
+            ):
 
-            with warnings.catch_warnings(record=True) as w:
-                entrypoints.init_all()
+                from numba.core import entrypoints
+                # Allow reinitialization
+                entrypoints._already_initialized = False
 
-            bad_str = "Numba extension module '_test_numba_bad_extension'"
-            for x in w:
-                if bad_str in str(x):
-                    break
-            else:
-                raise ValueError("Expected warning message not found")
+                with warnings.catch_warnings(record=True) as w:
+                    entrypoints.init_all()
 
-            # was our init function called?
-            self.assertEqual(counters['init'], 1)
+                bad_str = "Numba extension module '_test_numba_bad_extension'"
+                for x in w:
+                    if bad_str in str(x):
+                        break
+                else:
+                    raise ValueError("Expected warning message not found")
+
+                # was our init function called?
+                mod.init_func.assert_called_once()
 
         finally:
             # remove fake module
@@ -188,26 +175,23 @@ def box_dummy(typ, val, c):
             # will remove this module at the end of the test
             sys.modules[mod.__name__] = mod
 
-            # We are registering an entry point using the "numba" package
-            # ("distribution" in pkg_resources-speak) itself, though these are
-            # normally registered by other packages.
-            dist = "numba"
-            entrypoints = pkg_resources.get_entry_map(dist)
-            my_entrypoint = pkg_resources.EntryPoint(
-                "init",  # name of entry point
-                mod.__name__,  # module with entry point object
-                attrs=['init_func'],  # name of entry point object
-                dist=pkg_resources.get_distribution(dist)
+            my_entrypoint = importlib_metadata.EntryPoint(
+                'init',
+                '_test_numba_init_sequence:init_func',
+                'numba_extensions',
             )
-            entrypoints.setdefault('numba_extensions',
-                                   {})['init'] = my_entrypoint
-
-            @njit
-            def foo(x):
-                return x
 
-            ival = _DummyClass(10)
-            foo(ival)
+            with mock.patch.object(
+                importlib_metadata,
+                'entry_points',
+                return_value={'numba_extensions': (my_entrypoint,)},
+            ):
+                @njit
+                def foo(x):
+                    return x
+
+                ival = _DummyClass(10)
+                foo(ival)
         finally:
             # remove fake module
             if mod.__name__ in sys.modules: