Repository: pyinstaller/pyinstaller

Issue:
pkg_resources.DistributionNotFound on 4.4
I'm using PyInstaller with [azurerm](https://pypi.org/project/azurerm/) module in my company, we initially had an issue that was solved by submitting a hook [hook-azurerm.py](https://github.com/pyinstaller/pyinstaller-hooks-contrib/blob/master/src/_pyinstaller_hooks_contrib/hooks/stdhooks/hook-azurerm.py) to collect the metadata of the module.

Everything worked until PyInstaller 4.4 was released, this hook stopped working and we are getting:
```
Traceback (most recent call last):
  File "azure_test.py", line 3, in <module>
    print(azurerm.get_user_agent())
  File "azurerm/restfns.py", line 12, in get_user_agent
  File "pkg_resources/__init__.py", line 884, in require
  File "pkg_resources/__init__.py", line 770, in resolve
pkg_resources.DistributionNotFound: The 'requests' distribution was not found and is required by azurerm
```

I tied using 
`datas, binaries, hiddenimports = collect_all("azurerm")`
instead of `copy_metadata` with the same results.

Code to reproduce:
```
import azurerm
print(azurerm.get_user_agent())
```

Is there anything else that is needed for `pkg_resources` to work? 
We currently reverted to use PyInstaller 4.3.

Python: 3.7.9
PyInstaller: 4.4
OS: macOS BigSur (same on other platforms)



Discussion:
It'll be #5774 which broke it. I say *broke*, really it fixed it but a side effect of it being originally broken was that it in turn broke `pkg_reources` in a way that made `pkg_resources.require()` become almost a no-op which masked the fact that the metadata was missing. To appease `pkg_resources.require("azurem")` you need to recursively collect metadata for all dependencies of azurem for which you can either use [--recurive-copy-metadata=azurem](https://pyinstaller.readthedocs.io/en/stable/usage.html#cmdoption-recursive-copy-metadata) from the command line or [datas = copy_metadata("azurem", recursive=True)](https://pyinstaller.readthedocs.io/en/stable/hooks.html#PyInstaller.utils.hooks.copy_metadata) in either the spec file or the hook for azurem.
Does this mean `recursive` should be on by default for`copy_metadata`?
No, it's only needed for `pkg_resources.require()` and the `importlib.metadata` equivalent. `pkg_resources.get_distribution()` and `importlib.metadata.distribution()` and anything involving entry points don't require dependencies' metadata.
OK, guess we'll be fixing our hooks as-we-go, then, based on reports on what stopped working after 4.4.
That said, changing [this line](https://github.com/gbowerman/azurerm/blob/main/azurerm/restfns.py#L12) to:
```python
version = pkg_resources.get_distribution("azurerm").version
```
would remove the need for all that extra metadata.
> OK, guess we'll be fixing our hooks as-we-go, then, based on reports on what stopped working after 4.4.

Well the bytecode scanner should have detected the `pkg_resources.require()` and added the metadata accordingly which is why I'm marking this as a bug here rather than in the hook.
@bwoodsend Thanks, that actually worked.
I'm not sure if the `azurerm` module is still maintained but I can submit a PR and see, I will also submit a fix to the `azurerm` hook.
Hmph, it's a stupid one.
https://github.com/pyinstaller/pyinstaller/blob/3d5fa8caf1462637b5d604d638af282305502234/PyInstaller/depend/analysis.py#L811-L812
Should be:
```diff
         for method in recursive_methods:
-            need_metadata.update(bytecode.any_alias(package + "." + method))
+            need_recursive_metadata.update(bytecode.any_alias(package + "." + method))
 ```
So editing that one line fixes it. But guess we need more tests to cover that one.

Resolving patch:
diff --git a/PyInstaller/depend/analysis.py b/PyInstaller/depend/analysis.py
--- a/PyInstaller/depend/analysis.py
+++ b/PyInstaller/depend/analysis.py
@@ -809,7 +809,7 @@ def _metadata_from(self, package, methods=(), recursive_methods=()) -> set:
         for method in methods:
             need_metadata.update(bytecode.any_alias(package + "." + method))
         for method in recursive_methods:
-            need_metadata.update(bytecode.any_alias(package + "." + method))
+            need_recursive_metadata.update(bytecode.any_alias(package + "." + method))
 
         out = set()
 


Test patch:
diff --git a/tests/unit/test_pyimodulegraph.py b/tests/unit/test_pyimodulegraph.py
--- a/tests/unit/test_pyimodulegraph.py
+++ b/tests/unit/test_pyimodulegraph.py
@@ -208,3 +208,31 @@ def test_collect_rthooks_fail_1(tmpdir, monkeypatch):
     hd1 = _gen_pseudo_rthooks("h1", rh1, tmpdir, False)
     with pytest.raises(AssertionError):
         FakePyiModuleGraph(HOMEPATH, user_hook_dirs=[str(hd1)])
+
+
+class FakeGraph(analysis.PyiModuleGraph):
+    """A simplified module graph containing a single node module *foo* with user-defined content."""
+    def __init__(self, source):
+        self.code = compile(source, "<>", "exec")
+
+    def get_code_using(self, package):
+        return {"foo": self.code}
+
+
+def test_metadata_searching():
+    """Test the top level for bytecode scanning for metadata requirements."""
+    from PyInstaller.utils.hooks import copy_metadata
+
+    # This test analyses code which implies that PyInstaller's own metadata (and possibly that of its dependencies) is
+    # required.
+    pyinstaller = set(copy_metadata("pyinstaller"))
+    with_dependencies = set(copy_metadata("pyinstaller", recursive=True))
+
+    self = FakeGraph("from importlib.metadata import distribution; distribution('pyinstaller')")
+    assert pyinstaller == self.metadata_required()
+
+    self = FakeGraph("import pkg_resources; pkg_resources.get_distribution('pyinstaller')")
+    assert pyinstaller == self.metadata_required()
+
+    self = FakeGraph("import pkg_resources; pkg_resources.require('pyinstaller')")
+    assert with_dependencies == self.metadata_required()