Repository: pyinstaller/pyinstaller-hooks-contrib (language: python)

Issue:
A problem converting a grib2 file to a netCDF file using cfgrib and xarray
My test code - 
```
import cfgrib
import xarray as xr

# Path to the GRIB2 file
grib_file = 'GFS_Wind_000.grib2'

# Path for the output netCDF file
netcdf_file = 'output.nc'

# Open the GRIB2 file using cfgrib
ds = cfgrib.open_dataset(grib_file)

# Convert the GRIB2 dataset to xarray dataset
xds = xr.Dataset.from_dataframe(ds.to_dataframe())

# Save the xarray dataset to netCDF format
xds.to_netcdf(netcdf_file)
```

this works in python from the terminal window, but when I compile the code using the following pyinstaller command line -
```
pyinstaller --onefile --copy-metadata numpy --collect-data gribapi TryGrib.py
```
and then run the exe I get the following error message - 
```
Traceback (most recent call last):
  File "TryGrib.py", line 11, in <module>
    ds = cfgrib.open_dataset(grib_file)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "cfgrib\xarray_store.py", line 39, in open_dataset
  File "xarray\backends\api.py", line 557, in open_dataset
  File "xarray\backends\plugins.py", line 205, in get_backend
ValueError: unrecognized engine cfgrib must be one of: ['netcdf4', 'scipy', 'store']
```

any help will be greatly appreciated




Discussion / hints:
Looks like plugins are registed via `xarray.backends` entry point, so that needs to be collected.

For now, try creating a custom hook that will override the one from [`pyinstaller-hooks-contrib`](https://github.com/pyinstaller/pyinstaller-hooks-contrib/blob/master/src/_pyinstaller_hooks_contrib/hooks/stdhooks/hook-xarray.py); i.e., create an `extra-hooks` directory, and inside it, file called `hook-xarray.py`, with the following contents:

```python
# hook-xarray.py
from PyInstaller.utils.hooks import copy_metadata, collect_entry_point

# Collect additional backend plugins
datas, hiddenimports = collect_entry_point('xarray.backends')

# `xarray` requires `numpy` metadata due to several calls to its `xarray.core.utils.module_available` with specified
# `minversion` argument, which end up calling `importlib.metadata.version`.
datas += copy_metadata('numpy')
``` 

Then add `--additional-hooks-dir extra-hooks` to your PyInstaller command.


@rokm 
Thank you for your quick response
I did the following - 
1.  created a subfolder called "extra-hooks"
2.  in that folder I created a file with the name "hook-xarray.py" and I copied your code into this file
3.  when I compiled I issued the following command 
     pyinstaller --onefile --additional-hooks-dir extra-hooks TryGrib.py (the name of my short test script)
4.  the TryGrib.exe was created in the \dist folder
5.  when I run the TryGrib.exe I get the following message in the terminal window - 
(VE) PS C:\Data\GitHub\SearchWell\dist> .\TryGrib.exe     
Traceback (most recent call last):
  File "TryGrib.py", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "PyInstaller\loader\pyimod02_importers.py", line 419, in exec_module
  File "cfgrib\__init__.py", line 20, in <module>
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "PyInstaller\loader\pyimod02_importers.py", line 419, in exec_module
  File "cfgrib\cfmessage.py", line 29, in <module>
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "PyInstaller\loader\pyimod02_importers.py", line 419, in exec_module
  File "cfgrib\messages.py", line 28, in <module>
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "PyInstaller\loader\pyimod02_importers.py", line 419, in exec_module
  File "eccodes\__init__.py", line 13, in <module>
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "PyInstaller\loader\pyimod02_importers.py", line 419, in exec_module
  File "eccodes\eccodes.py", line 12, in <module>
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "PyInstaller\loader\pyimod02_importers.py", line 419, in exec_module
  File "gribapi\__init__.py", line 13, in <module>
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "PyInstaller\loader\pyimod02_importers.py", line 419, in exec_module
  File "gribapi\gribapi.py", line 34, in <module>
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "PyInstaller\loader\pyimod02_importers.py", line 419, in exec_module
  File "gribapi\errors.py", line 16, in <module>
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "PyInstaller\loader\pyimod02_importers.py", line 419, in exec_module
  File "gribapi\bindings.py", line 41, in <module>
  File "pkgutil.py", line 640, in get_data
  File "PyInstaller\loader\pyimod02_importers.py", line 228, in get_data
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\Dell\\AppData\\Local\\Temp\\_MEI152482\\gribapi\\grib_api.h'
[13404] Failed to execute script 'TryGrib' due to unhandled exception!
You omitted `--collect-data gribapi` that you had in your original PyInstaller command; you need that to collect that `gribapi\grib_api.h` file. So:

```
pyinstaller --onefile --additional-hooks-dir extra-hooks TryGrib.py --collect-data gribapi
```
@rokm 
With great gratitude, I salute you and your knowledge.
I have spent many hours on this ...... thank you very very much

Fix patch:
diff --git a/news/744.new.rst b/news/744.new.rst
new file mode 100644
index 000000000..557336f37
--- /dev/null
+++ b/news/744.new.rst
@@ -0,0 +1,3 @@
+Add hook for ``gribapi`` package from ``eccodes`` dist, in order to
+collect bundled headers and ensure that the eccodes shared library is
+collected from the build environment.
diff --git a/news/744.update.rst b/news/744.update.rst
new file mode 100644
index 000000000..5cbfc5339
--- /dev/null
+++ b/news/744.update.rst
@@ -0,0 +1,2 @@
+Extend the ``xarray`` hook to collect additional backend plugins that are
+registered via the ``xarray.backends`` entry-point (e.g., ``cfgrib``).
diff --git a/src/_pyinstaller_hooks_contrib/hooks/stdhooks/hook-gribapi.py b/src/_pyinstaller_hooks_contrib/hooks/stdhooks/hook-gribapi.py
new file mode 100644
index 000000000..c1c21a5c6
--- /dev/null
+++ b/src/_pyinstaller_hooks_contrib/hooks/stdhooks/hook-gribapi.py
@@ -0,0 +1,42 @@
+# ------------------------------------------------------------------
+# Copyright (c) 2024 PyInstaller Development Team.
+#
+# This file is distributed under the terms of the GNU General Public
+# License (version 2.0 or later).
+#
+# The full license is available in LICENSE.GPL.txt, distributed with
+# this software.
+#
+# SPDX-License-Identifier: GPL-2.0-or-later
+# ------------------------------------------------------------------
+
+import os
+
+from PyInstaller.utils.hooks import collect_data_files, get_module_attribute, logger
+
+# Collect the headers (eccodes.h, gribapi.h) that are bundled with the package.
+datas = collect_data_files('gribapi')
+
+# Collect the eccodes shared library
+binaries = []
+try:
+    library_path = get_module_attribute('gribapi.bindings', 'library_path')
+except Exception:
+    logger.warning("hook-gribapi: failed to query gribapi.bindings.library_path!", exc_info=True)
+    library_path = None
+
+if library_path:
+    if not os.path.isabs(library_path):
+        from PyInstaller.depend.utils import _resolveCtypesImports
+        resolved_binary = _resolveCtypesImports([os.path.basename(library_path)])
+        if resolved_binary:
+            library_path = resolved_binary[0][1]
+        else:
+            logger.warning("hook-gribapi: failed to resolve shared library name %r!", library_path)
+            library_path = None
+else:
+    logger.warning("hook-gribapi: could not determine path to eccodes shared library!")
+
+if library_path:
+    logger.debug("hook-gribapi: collecting eccodes shared library: %r", library_path)
+    binaries.append((library_path, '.'))
diff --git a/src/_pyinstaller_hooks_contrib/hooks/stdhooks/hook-xarray.py b/src/_pyinstaller_hooks_contrib/hooks/stdhooks/hook-xarray.py
index ac62596bd..fa41cd5a0 100644
--- a/src/_pyinstaller_hooks_contrib/hooks/stdhooks/hook-xarray.py
+++ b/src/_pyinstaller_hooks_contrib/hooks/stdhooks/hook-xarray.py
@@ -10,8 +10,11 @@
 # SPDX-License-Identifier: GPL-2.0-or-later
 # ------------------------------------------------------------------
 
-from PyInstaller.utils.hooks import copy_metadata
+from PyInstaller.utils.hooks import copy_metadata, collect_entry_point
+
+# Collect additional backend plugins that are registered via `xarray.backends` entry-point.
+datas, hiddenimports = collect_entry_point('xarray.backends')
 
 # `xarray` requires `numpy` metadata due to several calls to its `xarray.core.utils.module_available` with specified
 # `minversion` argument, which end up calling `importlib.metadata.version`.
-datas = copy_metadata('numpy')
+datas += copy_metadata('numpy')


Test patch:
diff --git a/.github/workflows/pr-test.yml b/.github/workflows/pr-test.yml
index 73ed23334..d4fdb4a0b 100644
--- a/.github/workflows/pr-test.yml
+++ b/.github/workflows/pr-test.yml
@@ -43,9 +43,11 @@ jobs:
         run: |
           git fetch origin ${{ github.base_ref }}
           set +e
-          diff --changed-group-format='%>' --unchanged-group-format='' \
+          # NOTE: we need to be compatible with both GNU diff and Apple/FreeBSD diff
+          diff \
             <(git show origin/${{ github.base_ref }}:requirements-test-libraries.txt) \
             <(git show HEAD:requirements-test-libraries.txt) \
+            | grep -E "^>" | sed "s/^> //" \
           > requirements-test-libraries.txt
           set -e
           echo '-r requirements-test.txt' >> requirements-test-libraries.txt
@@ -100,6 +102,8 @@ jobs:
           # These are dependencies of gmsh
           sudo apt-get install -y libglu1 libgl1 libxrender1 libxcursor1 libxft2 \
             libxinerama1 libgomp1
+          # This one is required by eccodes
+          sudo apt-get install -y libeccodes0
 
       - name: Install brew dependencies
         if: startsWith(matrix.os, 'macos')
@@ -112,6 +116,8 @@ jobs:
           brew install libdiscid
           # Install lsl library for pylsl
           brew install labstreaminglayer/tap/lsl
+          # This one is required by eccodes
+          brew install eccodes
 
       - name: Install dependencies
         shell: bash
diff --git a/requirements-test-libraries.txt b/requirements-test-libraries.txt
index a64a5261b..629c6673a 100644
--- a/requirements-test-libraries.txt
+++ b/requirements-test-libraries.txt
@@ -181,6 +181,9 @@ schwifty==2024.5.3
 
 # ------------------- Platform (OS) specifics
 
+# eccodes package requires the eccodes shared library provided by the environment (linux distribution, homebrew, or Anaconda).
+eccodes==1.7.0; sys_platform == 'darwin' or sys_platform == 'linux'
+
 # PyEnchant only pre-builds macOS and Windows
 pyenchant==3.2.2; sys_platform == 'darwin' or sys_platform == 'win32'
 
diff --git a/src/_pyinstaller_hooks_contrib/tests/test_libraries.py b/src/_pyinstaller_hooks_contrib/tests/test_libraries.py
index 424ba8e53..d44eaa420 100644
--- a/src/_pyinstaller_hooks_contrib/tests/test_libraries.py
+++ b/src/_pyinstaller_hooks_contrib/tests/test_libraries.py
@@ -2036,3 +2036,24 @@ def test_schwifty(pyi_builder):
         print(iban.bank_code)
         print(iban.account_code)
     """)
+
+
+@importorskip('gribapi')
+def test_eccodes_gribapi(pyi_builder):
+    pyi_builder.test_source("""
+        import sys
+        import os
+
+        # Basic import test
+        import gribapi
+
+        # Ensure that the eccodes shared library is bundled with the frozen application.
+        import gribapi.bindings
+
+        lib_filename = os.path.join(
+            sys._MEIPASS,
+            os.path.basename(gribapi.bindings.library_path),
+        )
+
+        assert os.path.isfile(lib_filename), f"Shared library {lib_filename!s} not found!"
+    """)