Repository: pyinstaller/pyinstaller

Issue:
No metadata path found for distribution 'greenlet' when importing gevent
## Description of the issue
Run pyinstaller as usual for a program that depends on `gevent`. Pyinstaller eventually fails (see stack trace below)

### Context information (for bug reports)

* Output of `pyinstaller --version`: ```4.10```
* Version of Python: 3.8.10
* Platform: Ubuntu 20.04
* How you installed Python: apt
* Did you also try this on another platform? Does it work there? Yes, on another similar computer with the same OS and it also fails.


* try the latest development version. Result: Same thing.


* follow *all* the instructions in our "If Things Go Wrong" Guide
  (https://github.com/pyinstaller/pyinstaller/wiki/If-Things-Go-Wrong): Does not apply

### Make sure [everything is packaged correctly](https://github.com/pyinstaller/pyinstaller/wiki/How-to-Report-Bugs#make-sure-everything-is-packaged-correctly)

Does not apply.


### A minimal example program which shows the error

```
import gevent
help(gevent)
```


### Stacktrace / full error message

Result of `python3.8 -m PyInstaller --onefile test.py --log-level DEBUG > out.txt 2>&1`:
[out.txt](https://github.com/pyinstaller/pyinstaller/files/8316518/out.txt)


### Additional info

I dug a bit in the source code and found these lines in `/usr/local/lib/python3.8/dist-packages/PyInstaller/utils/hooks/__init__.py`:
```
    if egg_path is None:
        # According to older implementations of this function, packages may have no metadata. I have no idea how this
        # can happen...
        raise RuntimeError(f"No metadata path found for distribution '{project_name}'.")
```

So two things:
- I have a system which could shed some light on this
- the result of `type(pkg_resources.get_distribution('greenlet').egg_info)` is indeed `<class 'NoneType'>`


Discussion:
I edited the original report where I said it works on another computer. It doesn't.

What happens is that my real program, which is quite large, does not import `gevent`, but for some reason, pyinstaller seems to need it on one system but not on the other. Maybe because of different versions of imported packages, who knows. But it doesn't matter for the main issue here I think.
Also, even though `pkg_resources.get_distribution('greenlet').egg_info` is `None`, the actual `.egg-info` file actually exists:

```
$ ls -al /usr/lib/python3/dist-packages/greenlet-0.4.15.egg-info 
-rw-r--r-- 1 root root 3763 Mar 14  2020 /usr/lib/python3/dist-packages/greenlet-0.4.15.egg-info
```
Looks like that `egg_info` field ends up being `None`  for all Ubuntu-packaged python packages where the corresponding `/usr/lib/python3/dist-packages/name-version.egg-info` is a file...
> What happens is that my real program, which is quite large, does not import `gevent`, but for some reason, pyinstaller seems to need it on one system but not on the other.

`gevent` is probably an optional dependency of some other package that your program uses, so it ends up being pulled in only if it is available in the environment (it could also be an optional dependency of an optional dependency that's present only on one system). Either way, it boils down to the difference in what's installed in the build environment. I'd advise you against using your base system python to build your program, as it might result in quite a bloated build (because you likely have more packages installed there than your program really needs, so optional dependencies will end up collected). Instead, create a virtual environment with minimal set of dependencies installed in via `pip install`. Or, if you really prefer Ubuntu-packaged versions of python packages, use a Docker container to set up a minimal environment.

---

But the case of missing metadata should probably be sorted out regardless of the above...
Yes I actually build the final executable on the target machine itself, which is as lean as it can be, but does have a different set of software installed. I should give virtual environments a try though.

---

My dev machine is on Fedora 35 and for `pkg_resources.get_distribution('greenlet').egg_info` I get this:
`'/usr/local/lib64/python3.10/site-packages/greenlet-1.1.2.dist-info'`
which is indeed a directory:

```
$ ls -al /usr/local/lib64/python3.10/site-packages/greenlet-1.1.2.dist-info
total 36
drwxr-xr-x. 1 root root  150 Jan 11 08:59 .
drwxr-xr-x. 1 root root 2186 Jan 12 15:34 ..
-rw-r--r--. 1 root root  849 Jan 11 08:59 AUTHORS
-rw-r--r--. 1 root root    4 Jan 11 08:59 INSTALLER
-rw-r--r--. 1 root root 1434 Jan 11 08:59 LICENSE
-rw-r--r--. 1 root root 2424 Jan 11 08:59 LICENSE.PSF
-rw-r--r--. 1 root root 3881 Jan 11 08:59 METADATA
-rw-r--r--. 1 root root 6120 Jan 11 08:59 RECORD
-rw-r--r--. 1 root root    0 Jan 11 08:59 REQUESTED
-rw-r--r--. 1 root root    9 Jan 11 08:59 top_level.txt
-rw-r--r--. 1 root root  152 Jan 11 08:59 WHEEL
```

Anyway from what I gather, `_copy_metadata_dest()` assumes `egg_info` is a directory, not a file.

It seems to me that PyInstaller should handle egg_info when it's a file, but also `pkg_resource.get_distribution().egg_info` should return something in that case. There nothing about a `Distribution.egg_info` property in [pkg_distribution help page](https://setuptools.pypa.io/en/latest/pkg_resources.html) and the only place `egg_info` is referenced as a parameter seems to imply it is a directory.
The egg-info isn't as well defined as the dist-info (which has its own PEP) but I'm very much leaning towards saying that Ubuntu is at fault here for a) using the obsolete egg format and b) making it a format which `pkg_resources` (a part of `setuptools` which is who sets the definition of an egg distribution since `setuptools` is normally what generates them in the first place) doesn't understand. 

Resolving patch:
diff --git a/PyInstaller/building/build_main.py b/PyInstaller/building/build_main.py
--- a/PyInstaller/building/build_main.py
+++ b/PyInstaller/building/build_main.py
@@ -262,6 +262,14 @@ def __init__(
         logger.info('Extending PYTHONPATH with paths\n' + pprint.pformat(self.pathex))
         sys.path.extend(self.pathex)
 
+        # If pkg_resources has already been imported, force update of its working set to account for changes made to
+        # sys.path. Otherwise, distribution data in the added path(s) may not be discovered.
+        if 'pkg_resources' in sys.modules:
+            # See https://github.com/pypa/setuptools/issues/373
+            import pkg_resources
+            if hasattr(pkg_resources, '_initialize_master_working_set'):
+                pkg_resources._initialize_master_working_set()
+
         # Set global variable to hold assembly binding redirects
         CONF['binding_redirects'] = []
 
diff --git a/PyInstaller/utils/hooks/__init__.py b/PyInstaller/utils/hooks/__init__.py
--- a/PyInstaller/utils/hooks/__init__.py
+++ b/PyInstaller/utils/hooks/__init__.py
@@ -889,9 +889,22 @@ def copy_metadata(package_name, recursive=False):
         package_name = todo.pop()
         if package_name in done:
             continue
+
         dist = pkg_resources.get_distribution(package_name)
-        dest = _copy_metadata_dest(dist.egg_info, dist.project_name)
-        out.append((dist.egg_info, dest))
+        if dist.egg_info is not None:
+            # If available, dist.egg_info points to the source .egg-info or .dist-info directory.
+            dest = _copy_metadata_dest(dist.egg_info, dist.project_name)
+            out.append((dist.egg_info, dest))
+        else:
+            # When .egg-info is not a directory but a single file, dist.egg_info is None, and we need to resolve the
+            # path ourselves. This format is common on Ubuntu/Debian with their deb-packaged python packages.
+            dist_src = _resolve_legacy_metadata_path(dist)
+            if dist_src is None:
+                raise RuntimeError(
+                    f"No metadata path found for distribution '{dist.project_name}' (legacy fallback search failed)."
+                )
+            out.append((dist_src, '.'))  # It is a file, so dest path needs to be '.'
+
         if not recursive:
             return out
         done.add(package_name)
@@ -904,6 +917,41 @@ def _normalise_dist(name: str) -> str:
     return name.lower().replace("_", "-")
 
 
+def _resolve_legacy_metadata_path(dist):
+    """
+    Attempt to resolve the legacy metadata file for the given distribution.
+    The .egg-info file is commonly used by Debian/Ubuntu when packaging python packages.
+
+    Args:
+        dist:
+            The distribution information as returned by ``pkg_resources.get_distribution("xyz")``.
+    Returns:
+        The path to the distribution's metadata file.
+    """
+
+    candidates = [
+        # This fallback was in place in pre-#5774 times. However, it is insufficient, because dist.egg_name() may be
+        # greenlet-0.4.15-py3.8 (Ubuntu 20.04 package) while the file we are searching for is greenlet-0.4.15.egg-info.
+        f"{dist.egg_name()}.egg-info",
+        # The extra name-version.egg-info path format
+        f"{dist.project_name}-{dist.version}.egg-info",
+        # And the name_with_underscores-version.egg-info.format
+        f"{dist.project_name.replace('-', '_')}-{dist.version}.egg-info",
+    ]
+
+    # As an additional attempt, try to remove the-pyX.Y suffix from egg name.
+    pyxx_suffix = f"-py{sys.version_info[0]}.{sys.version_info[1]}"
+    if dist.egg_name().endswith(pyxx_suffix):
+        candidates.append(dist.egg_name()[:-len(pyxx_suffix)] + ".egg-info")
+
+    for candidate in candidates:
+        candidate_path = os.path.join(dist.location, candidate)
+        if os.path.isfile(candidate_path):
+            return candidate_path
+
+    return None
+
+
 def _copy_metadata_dest(egg_path: str, project_name: str) -> str:
     """
     Choose an appropriate destination path for a distribution's metadata.


Test patch:
diff --git a/tests/functional/modules/pyi_single_file_metadata/README b/tests/functional/modules/pyi_single_file_metadata/README
new file mode 100644
--- /dev/null
+++ b/tests/functional/modules/pyi_single_file_metadata/README
@@ -0,0 +1,4 @@
+Package metadata stored in a single-file .egg-info format, as commonly
+found in Debian/Ubuntu packages. Used by test_misc::test_single_file_metadata
+which adds this directory to its search path.
+
diff --git a/tests/functional/modules/pyi_single_file_metadata/my_test_package-1.0.egg-info b/tests/functional/modules/pyi_single_file_metadata/my_test_package-1.0.egg-info
new file mode 100644
--- /dev/null
+++ b/tests/functional/modules/pyi_single_file_metadata/my_test_package-1.0.egg-info
@@ -0,0 +1,14 @@
+Metadata-Version: 2.1
+Name: my-test-package
+Version: 1.0
+Summary: Test package for single-file metadata
+Author: PyInstaller Team
+License: MIT
+Platform: Windows
+Platform: Linux
+Platform: Solaris
+Platform: Mac OS-X
+Platform: Unix
+Requires-Python: >=3.6
+
+The package for single-file metadata.
diff --git a/tests/functional/test_misc.py b/tests/functional/test_misc.py
--- a/tests/functional/test_misc.py
+++ b/tests/functional/test_misc.py
@@ -132,3 +132,25 @@ def test_onefile_cleanup_symlinked_dir(pyi_builder, tmpdir):
     for idx in range(5):
         output_file = os.path.join(output_dir, f'output-{idx}.txt')
         assert os.path.isfile(output_file)
+
+
+# Test that single-file metadata (as commonly found in Debian/Ubuntu packages) is properly collected by copy_metadata().
+def test_single_file_metadata(pyi_builder):
+    # Add directory containing the my-test-package metadata to search path
+    extra_path = os.path.join(_MODULES_DIR, "pyi_single_file_metadata")
+
+    pyi_builder.test_source(
+        """
+        import pkg_resources
+
+        # The pkg_resources.get_distribution() call automatically triggers collection of the metadata. While it does not
+        # raise an error if metadata is not found while freezing, the calls below will fall at run-time in that case.
+        dist = pkg_resources.get_distribution('my-test-package')
+
+        # Sanity check
+        assert dist.project_name == 'my-test-package'
+        assert dist.version == '1.0'
+        assert dist.egg_name() == f'my_test_package-{dist.version}-py{sys.version_info[0]}.{sys.version_info[1]}'
+        """,
+        pyi_args=['--paths', extra_path]
+    )