Repository: ESMValGroup/ESMValCore

Issue:
`pkg_resources` is deprecated
The `pkg_resources` package used in `esmvalcore/_main.py` is [deprecated](https://setuptools.pypa.io/en/latest/pkg_resources.html). Therefore the function `pkg_resources.iter_entry_points` needs to be replaced with [`importlib.metadata.entry_points`](https://docs.python.org/3/library/importlib.metadata.html#entry-points).

Discussion:
codecov[bot]: ## [Codecov](https://app.codecov.io/gh/ESMValGroup/ESMValCore/pull/2096?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ESMValGroup) Report
> Merging [#2096](https://app.codecov.io/gh/ESMValGroup/ESMValCore/pull/2096?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ESMValGroup) (5f49e8c) into [main](https://app.codecov.io/gh/ESMValGroup/ESMValCore/commit/179312e41f964bcb9cbd44f0ee234d62127d20ab?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ESMValGroup) (179312e) will **not change** coverage.
> The diff coverage is `100.00%`.

```diff
@@           Coverage Diff           @@
##             main    #2096   +/-   ##
=======================================
  Coverage   93.08%   93.08%           
=======================================
  Files         237      237           
  Lines       12793    12793           
=======================================
  Hits        11909    11909           
  Misses        884      884           
```


| [Impacted Files](https://app.codecov.io/gh/ESMValGroup/ESMValCore/pull/2096?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ESMValGroup) | Coverage Δ | |
|---|---|---|
| [esmvalcore/\_main.py](https://app.codecov.io/gh/ESMValGroup/ESMValCore/pull/2096?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ESMValGroup#diff-ZXNtdmFsY29yZS9fbWFpbi5weQ==) | `90.87% <100.00%> (ø)` | |


bouweandela: I'm not able to reproduce the issue reported by Codacy locally and usage with a `group` keyword argument is exactly as in the examples on the [docs](https://setuptools.pypa.io/en/latest/userguide/entry_point.html#entry-points) page, so I'm not sure what is causing it.
valeriupredoi: first clue as for the fail issue - there are two slightly different packages installed (for all Python versions, not just 3.9):
```
 + importlib-metadata                    6.7.0  pyha770c72_0             conda-forge/noarch        26kB
 + importlib_metadata                    6.7.0  hd8ed1ab_0               conda-forge/noarch         9kB
 ```
 this no bueno. Lemme dig deeper
valeriupredoi: so `importlib_metadata` is an old package that was decomissioned by the maintainer in January, see https://anaconda.org/anaconda/importlib_metadata - and they migrated it to `importlib-metadat` which is at 6.7.0 now (whereas _metadata stopped at 6.0.0). We have two issues here:

- we have a dep in our env that pulls `importlib_metadata`; we end up with both -metadata and _metadata; the import in module is obv with `import importlib_metadat` so that ends up importing the old package
- conda forge is silly and reports that _metadata is at the same version 6.7.0 as -metadata (most prob a bug in the ABI) - this needs reported
valeriupredoi: @zklaus what's the best way to report a broken package to conda-forge? Via gitter or open an actual issue? :beer: 
valeriupredoi: OK gotcha - iris needs both importlib_metadata and importlib-metadata (hopefully not via one of their deps)
valeriupredoi: > OK gotcha - iris needs both importlib_metadata and importlib-metadata (hopefully not via one of their deps)

it's in fact via `dask` and I stopped pruning environments and solving dep graphs there, I asked on Gitter first https://app.gitter.im/#/room/#conda-forge:matrix.org
bouweandela: Note that this module is part of the standard library: https://docs.python.org/3.11/library/importlib.metadata.html
valeriupredoi: ah then I chased a red herring then :fish: - still, for Python 3.9 and only for Python 3.9 there are two `importlib-/_resources`:
```
    + importlib-resources                  5.12.0  pyhd8ed1ab_0            conda-forge/noarch         9kB
    + importlib_metadata                    6.7.0  hd8ed1ab_0              conda-forge/noarch         9kB
    + importlib_resources                  5.12.0  pyhd8ed1ab_0            conda-forge/noarch        31kB
```

Look at the sizes of the pkgs - lemme try try not specifying `importlib_resources` in our env
valeriupredoi: boom! Problem solved - by good old conda-forge packaging :grin: If you want to you can add `importlib_metadata` as direct requirement for us, but I'd not really bother since that's coming from scipy via iris :+1: 
bouweandela: OK, but the point of this pull request was to use the Python standard library instead of the package that backports that to older Python versions. I'll have a look if I can get it to work for Python 3.9 when I have some time.
valeriupredoi: Why are you keen on using the standard library - as you can see from here, the c-f package is more stable, rather, I thought the idea was to retire pkg_resources 🍺 
zklaus: I'd like to explain a bit the ecosystem. Pypi decided that `-` and `_` are the same thing, i.e. in URLs, and when installing stuff with Pip you can always use either. Conda-forge does not work like that. If you want to support both formats, you need to provide two packages, which is relatively frequently done with multi-output recipes, i.e. a single recipe producing multiple packages. In that case, it usually creates one of the two packages as a dummy that depends on the other one. Here, [the recipe](https://github.com/conda-forge/importlib_metadata-feedstock/blob/3e30643929461f6a388c9084c864e1658e4fa144/recipe/meta.yaml) generates the real package as `importlib-metadata` and a dummy package `importlib_metadata` that depends on the former.

So when you depend on `importlib_metadata` you automatically also depend on `importlib-metadata`, but these are not distinct packages.

> ah then I chased a red herring then fish - still, for Python 3.9 and only for Python 3.9 there are two `importlib-/_resources`:

How did you come to this conclusion? The conda-forge packages are `noarch`, so there aren't different packages for different Python versions.


zklaus: > so `importlib_metadata` is an old package that was decomissioned by the maintainer in January, see https://anaconda.org/anaconda/importlib_metadata - [...]

You are looking in the wrong place. We don't care about Anaconda packages; the corresponding Conda-forge package is alive and well at https://anaconda.org/conda-forge/importlib_metadata.


zklaus: To record some insight we generated at the workshop yesterday: In Python 3.9, `entry_points()` does not have a `group` keyword and instead just returns all entry points as a Python `dict` keyed by the entry point groups. From Python 3.10 on, `entry_points` is more powerful and can do some filtering. It also does not return a `dict`, but a specialized `EntryPoints` object.

Both `dict` in Python 3.9 and `EntryPoints` in 3.10+ support `[]` access, so we could just use `entry_points()["emsvaltool_commands"]`, which will return all registered esmvaltool commands and raise a `KeyError` if there none.

The downside is that from 3.10 on, this usage is deprecated and will generate a `DeprecationWarning`; I could not find out when it will actually be removed.
zklaus: Should we really be removing `importlib_resources`? Note that this is a different thing than `importlib_metadata`. While `importlib_metadata` is used to access entry points, distribution package names, etc., `importlib_resources` is used to access resources, such as binary and text files like recipes in the distribution. Confusingly, both functionalities were provided by `pkg_resources`.

We should be using `importlib_resources` (also in the standard lib as `importlib.resources`) to load packaged recipes and configuration files, though we may not be doing that consequently at the moment. Basically, all uses of `__file__` should be replaced with some use of importlib resources. That is a different discussion, but I point it out here to ask whether we really want to remove the dependency on `importlib_resources` in this PR.
zklaus: PS: Seems the only place where this is used is at https://github.com/ESMValGroup/ESMValCore/blob/91aa9e4d274e8c1e04477ee3af65764e2b0e8d8f/esmvalcore/config/_config.py#L24, so if we consider Python 3.8 support to be dropped, we can remove the dependency, but then should also remove the special import code there, probably both together in a separate PR.
bouweandela: Yes, I forgot to remove that code when dropping support for Python 3.8.
valeriupredoi: python=3.8 went to the fishes a while back, Klaus :fish: 

Patch:
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index 31f3d57a8..2b01ae61c 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -50,5 +50,4 @@ repos:
       - id: mypy
         additional_dependencies:
           - 'types-PyYAML'
-          - 'types-pkg_resources'
           - 'types-requests'
diff --git a/environment.yml b/environment.yml
index b18c7f341..cb066cba0 100644
--- a/environment.yml
+++ b/environment.yml
@@ -19,7 +19,7 @@ dependencies:
   - fire
   - geopy
   - humanfriendly
-  - importlib_resources
+  - importlib_metadata  # required for Python < 3.10
   - iris >=3.6.0
   - iris-esmf-regrid >=0.7.0
   - isodate
@@ -64,7 +64,6 @@ dependencies:
   # Not on conda-forge - ESMValTool_sample_data==0.0.3
   # Still for testing, MyPy library stubs
   - types-requests
-  - types-pkg_resources
   - types-PyYAML
   # Python packages needed for installing in development mode
   - codespell
diff --git a/esmvalcore/_main.py b/esmvalcore/_main.py
index 0aee610ed..da5cdd026 100755
--- a/esmvalcore/_main.py
+++ b/esmvalcore/_main.py
@@ -29,10 +29,15 @@ http://docs.esmvaltool.org. Have fun!
 # pylint: disable=import-outside-toplevel
 import logging
 import os
+import sys
 from pathlib import Path
 
+if (sys.version_info.major, sys.version_info.minor) < (3, 10):
+    from importlib_metadata import entry_points
+else:
+    from importlib.metadata import entry_points  # type: ignore
+
 import fire
-from pkg_resources import iter_entry_points
 
 # set up logging
 logger = logging.getLogger(__name__)
@@ -299,12 +304,13 @@ class ESMValTool():
         self.config = Config()
         self.recipes = Recipes()
         self._extra_packages = {}
-        if not list(iter_entry_points('esmvaltool_commands')):
+        esmvaltool_commands = entry_points(group='esmvaltool_commands')
+        if not esmvaltool_commands:
             print("Running esmvaltool executable from ESMValCore. "
                   "No other command line utilities are available "
                   "until ESMValTool is installed.")
-        for entry_point in iter_entry_points('esmvaltool_commands'):
-            self._extra_packages[entry_point.dist.project_name] = \
+        for entry_point in esmvaltool_commands:
+            self._extra_packages[entry_point.dist.name] = \
                 entry_point.dist.version
             if hasattr(self, entry_point.name):
                 logger.error('Registered command %s already exists',
diff --git a/esmvalcore/config/_config.py b/esmvalcore/config/_config.py
index aaadb955e..85f8f1f1c 100644
--- a/esmvalcore/config/_config.py
+++ b/esmvalcore/config/_config.py
@@ -5,8 +5,8 @@ import collections.abc
 import fnmatch
 import logging
 import os
-import sys
 from functools import lru_cache
+from importlib.resources import files as importlib_files
 from pathlib import Path
 
 import yaml
@@ -21,12 +21,6 @@ TASKSEP = os.sep
 
 CFG = {}
 
-if sys.version_info[:2] >= (3, 9):
-    # pylint: disable=no-name-in-module
-    from importlib.resources import files as importlib_files
-else:
-    from importlib_resources import files as importlib_files
-
 
 def _deep_update(dictionary, update):
     for key, value in update.items():
diff --git a/setup.py b/setup.py
index b93292955..092379b6b 100755
--- a/setup.py
+++ b/setup.py
@@ -39,7 +39,7 @@ REQUIREMENTS = {
         'fire',
         'geopy',
         'humanfriendly',
-        "importlib_resources;python_version<'3.9'",
+        "importlib_metadata;python_version<'3.10'",
         'isodate',
         'jinja2',
         'nc-time-axis',  # needed by iris.plot
@@ -78,7 +78,6 @@ REQUIREMENTS = {
         # MyPy library stubs
         'mypy>=0.990',
         'types-requests',
-        'types-pkg_resources',
         'types-PyYAML',
     ],
     # Documentation dependencies


Test patch:
diff --git a/tests/unit/main/test_esmvaltool.py b/tests/unit/main/test_esmvaltool.py
index 0bbe1b850..42e981511 100644
--- a/tests/unit/main/test_esmvaltool.py
+++ b/tests/unit/main/test_esmvaltool.py
@@ -178,18 +178,18 @@ def test_do_not_clean_preproc_dir(session):
     assert session._fixed_file_dir.exists()
 
 
-@mock.patch('esmvalcore._main.iter_entry_points')
+@mock.patch('esmvalcore._main.entry_points')
 def test_header(mock_entry_points, caplog):
 
     entry_point = mock.Mock()
-    entry_point.dist.project_name = 'MyEntry'
+    entry_point.dist.name = 'MyEntry'
     entry_point.dist.version = 'v42.42.42'
     entry_point.name = 'Entry name'
     mock_entry_points.return_value = [entry_point]
     with caplog.at_level(logging.INFO):
         ESMValTool()._log_header(
             'path_to_config_file',
-            ['path_to_log_file1', 'path_to_log_file2']
+            ['path_to_log_file1', 'path_to_log_file2'],
         )
 
     assert len(caplog.messages) == 8
@@ -218,8 +218,10 @@ def test_get_recipe(is_file):
 @mock.patch('os.path.isfile')
 @mock.patch('esmvalcore.config._diagnostics.DIAGNOSTICS')
 def test_get_installed_recipe(diagnostics, is_file):
+
     def encountered(path):
         return Path(path) == Path('/install_folder/recipe.yaml')
+
     is_file.side_effect = encountered
     diagnostics.recipes = Path('/install_folder')
     recipe = ESMValTool()._get_recipe('recipe.yaml')