Repository: pyinstaller/pyinstaller

Issue:
Support pkgutil.iter_modules
I have a plugin system in Dosage which uses pkgutil.iter_modules (https://github.com/webcomics/dosage/blob/master/dosagelib/loader.py) to find all modules with plugins. It seems iter_modules is only supported for Python's built in loaders at the moment (see https://hg.python.org/cpython/file/tip/Lib/pkgutil.py#l128, https://hg.python.org/cpython/file/tip/Lib/pkgutil.py#l176 and https://hg.python.org/cpython/file/tip/Lib/pkgutil.py#l391). It would be nice if PyInstaller's FrozenImporter had a iter_modules implementing the pkgutil protocol¹, so one could discover modules in PyInstaller-packaged libraries.

(As you can see, I'm currently special-casing PyInstaller by accessing the "toc" set of FrozenImporter directly. This works, but feels a bit hacky to me...)

¹ This "contract" seems to be rather under-documented. It's implemented in pkgutil, which references PEP-302 a lot, but that does not mention iter_modules anywhere - I can only find a reference to this method in some [bug report](http://bugs.python.org/issue14982), [commit message](https://hg.python.org/cpython/rev/b92fa1c5a96b) and [withdrawn PEP-402](https://www.python.org/dev/peps/pep-0402/).

A Module Error Bug
Traceback (most recent call last):
  File "Movie Editor.py", line 4, in <module>
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "PyInstaller\loader\pyimod03_importers.py", line 540, in exec_module
  File "moviepy\editor.py", line 87, in <module>
  File "<string>", line 1, in <module>
AttributeError: module 'moviepy.audio.fx.all' has no attribute 'audio_fadein'
[14852] Failed to execute script Movie Editor


The Line 4 is: "import moviepy.editor"
The python file works all fine.

Why is it showing such a error???


Discussion:
Thanks a lot @TobiX !

I have been struggling to get a frozen django app work with the `run_gunicorn` command which is discovered by django using `pkgutil.iter_modules()` which fails to work as expected inside the frozen app. 

I tried your[ `toc` hack ](https://github.com/webcomics/dosage/blob/master/dosagelib/loader.py)and it seems to work. 

Right now I am stuck at a `pkg_resources.load_entry_point()` exception for `gunicorn`

```
Error: class uri 'simple' invalid or not found:

[Traceback (most recent call last):
  File "site-packages/gunicorn/util.py", line 125, in load_class
  File "site-packages/pkg_resources/__init__.py", line 568, in load_entry_point
  File "site-packages/pkg_resources/__init__.py", line 561, in get_distribution
  File "site-packages/pkg_resources/__init__.py", line 441, in get_provider
  File "site-packages/pkg_resources/__init__.py", line 962, in require
  File "site-packages/pkg_resources/__init__.py", line 849, in resolve
DistributionNotFound: The 'gunicorn' distribution was not found and is required by the application
]
```

Anyway, back to more digging. Thanks again for the `toc` hack @TobiX 

I was able to solve the `pkg_resources.load_entry_point()` issue by adding a `hooks.copy_metadata('gunicorn')`. Thanks to @bjones1 (https://github.com/pyinstaller/pyinstaller/issues/1713) 

```
from PyInstaller.utils import hooks
datas.extend(hooks.copy_metadata('gunicorn'))
```

hi,

having the same issue, I've spun up a test repo that shows that the above workarounds aren't working for me 

https://github.com/gendrion/pyinstaller_test

as you can see it prints the following when run through an interpreter

```
Start Imports
module names from iter_modules:  ['mymodule.file1', 'mymodule.file2']
module names after get_importer toc:  ['mymodule.file1', 'mymodule.file2']
File 1 Imported
File 2 Imported
End Imports
```

but when running the exe:

```
Start Imports
module names from iter_modules:  []
module names after get_importer toc:  []
End Imports
```

I would expect get_importer to get the files even though iter_modules can't when using it via pyinstaller

Please advise :)
Hi @gendrion, 
I have been having the same issue (workaround not working) on my own project and finally got your test to work (thanks for this btw).

Solution was to create a **hook-mymodule.py** file (at same level as myFile.py) containing:
```python
from PyInstaller.utils.hooks import collect_submodules
hiddenimports = collect_submodules('mymodule')
```
and specify the `--additional-hooks-dir=.` when you call pyinstaller (or set `hookspath=['.']` in a previously generated spec file).

(Ps. there's a typo in the `__init__.py` filename in your example which initially stumped me)
works fine !
Hi. update to pyinstaller 3.4 seem to have broken something:
- collect_submodule still works (as we can see in the log)
- but the [trick described here](https://github.com/gendrion/pyinstaller_test/blob/master/mymodule/__init.__py) using `pkgutil.get_importer` does not work anymore.

How can I list all submodule from a given point (a-la `pkgutil.iter_modules` which is known to not work with pyinstaller) ?
With the current PyInstaller (`develop` branch, but I guess it's the same for v3.4), indeed, `pkgutil.get_importer()` returns `None`. However, using `pkgutil.iter_importers(package.__name__)` with `package` being your main package seems to work. Here's what I ended up doing:

```python
toc = set()
for importer in pkgutil.iter_importers('qutebrowser'):
    if hasattr(importer, 'toc'):
        toc |= importer.toc
for name in toc:
    if name.startswith(components.__name__ + '.'):
        yield name
```

Full file: [loader.py](https://github.com/qutebrowser/qutebrowser/blob/master/qutebrowser/extensions/loader.py)
Just for the record: This was "broken" by b5c9f21fe6959d63dde77ee07ba60122e515f254 and can be restored by this diff:

```diff
--- a/PyInstaller/loader/pyimod03_importers.py
+++ b/PyInstaller/loader/pyimod03_importers.py
@@ -802,12 +802,12 @@ def install():
     fimp = FrozenImporter()
     sys.meta_path.append(fimp)
 
-    if sys.version_info[0] == 2:
-        # _`path_hook installation`: Add the FrozenImporter to `sys.path_hook`,
-        # too, since `pkgutil.get_loader()` does not use `sys.meta_path`. See
-        # issue #1689.
-        sys.path_hooks.append(fimp)
+    # _`path_hook installation`: Add the FrozenImporter to `sys.path_hook`,
+    # too, since `pkgutil.get_loader()` does not use `sys.meta_path`. See
+    # issue #1689.
+    sys.path_hooks.append(fimp)
 
+    if sys.version_info[0] == 2:
         # Import hook for renamed C extension modules. The path hook above means
         # that the standard Python import system for C extensions will only work
         # if those extensions are not in a submodule. In particular: A C
```

I'm a bit hesitant to suggest this as a fix, because I don't know the ins and outs of the python import system and how this might influence other behaviour.
Hi. It would be cool to keep this feature running, somehow, since I have a pyinstaller executable that relies on this "discovery" mecanism (basically it is able to find its sub command automatically, each subcommand being on single .py file)

Any updates regarding this issue?
I want to use the faker module and having the issue with dynamic loading (#3839).
Is there a way around this?
This is a problem for me too. It would be great if it could be fixed. In the meantime is there a suggested workaround / alternative?
I posted a workaround in my comment above.
Thanks @The-Compiler, I missed that before. I'll give it a try!
Here is my updated function after recommendation from @The-Compiler 👍 
```python

def iter_namespace(ns_pkg):
    """Pyinstaller-compatible namespace iteration.

    Yields the name of all modules found at a given Fully-qualified path.

    To have it running with pyinstaller, it requires to ensure a hook inject the
    "hidden" modules from your plugins folder inside the executable:

    - if your plugins are under the ``myappname/pluginfolder`` module
    - create a file ``specs/hook-<myappname.pluginfolder>.py``
    - content of this file should be:

        .. code-block:: python

            from PyInstaller.utils.hooks import collect_submodules
            hiddenimports = collect_submodules('<myappname.pluginfolder>')
    """
    prefix = ns_pkg.__name__ + "."
    for p in pkgutil.iter_modules(ns_pkg.__path__, prefix):
        yield p[1]

    # special handling when the package is bundled with PyInstaller 3.5
    # See https://github.com/pyinstaller/pyinstaller/issues/1905#issuecomment-445787510
    toc = set()
    for importer in pkgutil.iter_importers(ns_pkg.__name__.partition(".")[0]):
        if hasattr(importer, 'toc'):
            toc |= importer.toc
    for name in toc:
        if name.startswith(prefix):
            yield name
```

works with Python 3.6 and Pyinstaller 3.5
It's because [`moviepy.audio.fx.all` in moviepy 1.0.3](https://github.com/Zulko/moviepy/blob/v1.0.3/moviepy/audio/fx/all/__init__.py) is supposed to load all submodules using `pkgutil.iter_modules()`, which is currently unsupported by PyInstaller (looks like we have an open issue about that already, too - #1905). Therefore, `moviepy.audio.fx.all.audio_fadein`, which is supposed to be a submodule, does not exist, resulting in that `AttributeError`.

It seems `moviepy` changed that approach in their [git `master`](https://github.com/Zulko/moviepy/blob/master/moviepy/audio/fx/all/__init__.py), although the change is not part of any release yet. So I guess your best bet at the moment is trying to install `moviepy` from their git master. Or wait for new `moviepy` release. Or wait until someone adds support for pkgutil.iter_modules() to PyInstaller...
Or, a possible work-around would also be to build with `--collect-submodules moviepy --debug noarchive` arguments added to your PyInstaller command. This will collect all submodules from `moviepy` and also write all collected modules directly into application's directory instead of into embedded archive, which should make them discoverable by built-in importers used by `pkgutil.iter_modules`.

Thank You and please repair this problem...
This can be tracked on #1905.

Resolving patch:
diff --git a/PyInstaller/hooks/rthooks/pyi_rth_pkgutil.py b/PyInstaller/hooks/rthooks/pyi_rth_pkgutil.py
new file mode 100644
--- /dev/null
+++ b/PyInstaller/hooks/rthooks/pyi_rth_pkgutil.py
@@ -0,0 +1,91 @@
+#-----------------------------------------------------------------------------
+# Copyright (c) 2021, PyInstaller Development Team.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+#
+# The full license is in the file COPYING.txt, distributed with this software.
+#
+# SPDX-License-Identifier: Apache-2.0
+#-----------------------------------------------------------------------------
+#
+# This rthook overrides pkgutil.iter_modules with custom implementation
+# that uses PyInstaller's FrozenImporter to list sub-modules embedded
+# in the PYZ archive. The non-embedded modules (binary extensions, or
+# .pyc modules in noarchive build) are handled by original pkgutil
+# iter_modules implementation (and consequently, python's FileFinder).
+#
+# The preferred way of adding support for iter_modules would be adding
+# non-standard iter_modules() method to FrozenImporter itself. However,
+# that seems to work only for path entry finders (for use with sys.path_hooks),
+# while PyInstaller's FrozenImporter is registered as meta path finders
+# (for use with sys.meta_path). Turning FrozenImporter into path entry
+# finder, would seemingly require the latter to support on-filesystem
+# resources (e.g., extension modules) in addition to PYZ-embedded ones.
+#
+# Therefore, we instead opt for overriding pkgutil.iter_modules with
+# custom implementation that augments the output of original
+# implementation with contents of PYZ archive from FrozenImporter's TOC.
+
+
+import os
+import sys
+
+import pkgutil
+
+from pyimod03_importers import FrozenImporter
+
+
+SYS_PREFIX = sys._MEIPASS + os.path.sep
+SYS_PREFIXLEN = len(SYS_PREFIX)
+
+
+_orig_pkgutil_iter_modules = pkgutil.iter_modules
+
+
+def _pyi_pkgutil_iter_modules(path=None, prefix=''):
+    # Use original implementation to discover on-filesystem
+    # modules (binary extensions in regular builds, or both binary
+    # extensions and compiled pyc modules in noarchive debug builds)
+    yield from _orig_pkgutil_iter_modules(path, prefix)
+
+    # Find the instance of PyInstaller's FrozenImporter
+    for importer in pkgutil.iter_importers():
+        if isinstance(importer, FrozenImporter):
+            break
+    else:
+        return
+
+    if not path:
+        # Search for all top-level packages/modules. These will have
+        # no dots in their entry names
+        for entry in importer.toc:
+            if entry.count('.') != 0:
+                continue
+            is_pkg = importer.is_package(entry)
+            yield pkgutil.ModuleInfo(importer, prefix + entry, is_pkg)
+    else:
+        # Only single path is supported, and it must start with
+        # sys._MEIPASS
+        pkg_path = os.path.normpath(path[0])
+        assert pkg_path.startswith(SYS_PREFIX)
+        # Construct package prefix from path...
+        pkg_prefix = pkg_path[SYS_PREFIXLEN:]
+        pkg_prefix = pkg_prefix.replace(os.path.sep, '.')
+        # ... and ensure it ends with a dot (so we can directly filter
+        # out the package itself)
+        if not pkg_prefix.endswith('.'):
+            pkg_prefix += '.'
+        pkg_prefix_len = len(pkg_prefix)
+
+        for entry in importer.toc:
+            if not entry.startswith(pkg_prefix):
+                continue
+            name = entry[pkg_prefix_len:]
+            if name.count('.') != 0:
+                continue
+            is_pkg = importer.is_package(entry)
+            yield pkgutil.ModuleInfo(importer, prefix + name, is_pkg)
+
+
+pkgutil.iter_modules = _pyi_pkgutil_iter_modules


Test patch:
diff --git a/tests/functional/scripts/pyi_pkgutil_iter_modules.py b/tests/functional/scripts/pyi_pkgutil_iter_modules.py
new file mode 100644
--- /dev/null
+++ b/tests/functional/scripts/pyi_pkgutil_iter_modules.py
@@ -0,0 +1,40 @@
+#-----------------------------------------------------------------------------
+# Copyright (c) 2021, PyInstaller Development Team.
+#
+# Distributed under the terms of the GNU General Public License (version 2
+# or later) with exception for distributing the bootloader.
+#
+# The full license is in the file COPYING.txt, distributed with this software.
+#
+# SPDX-License-Identifier: (GPL-2.0-or-later WITH Bootloader-exception)
+#-----------------------------------------------------------------------------
+
+
+import sys
+import argparse
+import pkgutil
+import importlib
+
+# Argument parser
+parser = argparse.ArgumentParser(description="pkgutil iter_modules test")
+parser.add_argument('package', type=str, help="Package to test.")
+parser.add_argument('--prefix', type=str, default='',
+                    help="Optional prefix to pass to iter_modules.")
+parser.add_argument('--output-file', default=None, type=str,
+                    help="Output file.")
+args = parser.parse_args()
+
+# Output file (optional)
+if args.output_file:
+    fp = open(args.output_file, 'w')
+else:
+    fp = sys.stdout
+
+# Iterate over package's module
+package = importlib.import_module(args.package)
+for module in pkgutil.iter_modules(package.__path__, args.prefix):
+    print("%s;%d" % (module.name, module.ispkg), file=fp)
+
+# Cleanup
+if args.output_file:
+    fp.close()
diff --git a/tests/functional/test_pkgutil.py b/tests/functional/test_pkgutil.py
new file mode 100644
--- /dev/null
+++ b/tests/functional/test_pkgutil.py
@@ -0,0 +1,93 @@
+#-----------------------------------------------------------------------------
+# Copyright (c) 2021, PyInstaller Development Team.
+#
+# Distributed under the terms of the GNU General Public License (version 2
+# or later) with exception for distributing the bootloader.
+#
+# The full license is in the file COPYING.txt, distributed with this software.
+#
+# SPDX-License-Identifier: (GPL-2.0-or-later WITH Bootloader-exception)
+#-----------------------------------------------------------------------------
+#
+# Tests for pkgutil.iter_modules(). The test attempts to list contents
+# of a package in both unfrozen and frozen version, and compares the
+# obtained lists.
+#
+# We test two packages; altgraph (pure-python) and psutil (contains
+# binary extensions). The extensions are present on filesystem as-is,
+# and are therefore handled by python's FileFinder. The collected .pyc
+# modules, however, are embedded in PYZ archive, and are not visible
+# to standard python's finders/loaders. The exception to that is
+# noarchive mode, where .pyc modules are not collected into archive;
+# as they are present on filesystem as-is, they are again handled
+# directly by python's FileFinder. Therefore, the test is performed
+# both in archive and in noarchive mode, to cover both cases.
+
+
+import os
+
+import pytest
+
+from PyInstaller.compat import exec_python_rc
+
+
+# Read the output file produced by test script. Each line consists of
+# two elements separated by semi-colon: name;ispackage
+def _read_results_file(filename):
+    output = []
+    with open(filename, 'r') as fp:
+        for line in fp:
+            tokens = line.split(';')
+            assert len(tokens) == 2
+            output.append((tokens[0], int(tokens[1])))
+    # Sort the results, so we can compare them
+    return sorted(output)
+
+
+@pytest.mark.parametrize('package', [
+    'altgraph',  # pure python package
+    'psutil',  # package with extensions
+    'psutil.tests',  # sub-package
+])
+@pytest.mark.parametrize('archive', ['archive', 'noarchive'])
+def test_pkgutil_iter_modules(package, script_dir, tmpdir, pyi_builder,
+                              archive):
+    # Ensure package is available
+    pytest.importorskip(package)
+
+    # Full path to test script
+    test_script = 'pyi_pkgutil_iter_modules.py'
+    test_script = os.path.join(script_dir, test_script)
+
+    # Run unfrozen test script
+    out_unfrozen = os.path.join(tmpdir, 'output-unfrozen.txt')
+    rc = exec_python_rc(test_script, package, '--output-file', out_unfrozen)
+    assert rc == 0
+    # Read results
+    results_unfrozen = _read_results_file(out_unfrozen)
+
+    # Run frozen script
+    out_frozen = os.path.join(tmpdir, 'output-frozen.txt')
+    debug_args = ['--debug', 'noarchive'] if archive == 'noarchive' else []
+    pyi_builder.test_script(
+        test_script,
+        pyi_args=[
+            # ensure everything is collected
+            '--collect-submodules', package,
+            # however, psutil.tests pulls in pip and wheel, which in
+            # turn manage to pull in pyinstaller.exe/__main__ and break
+            # Windows noarchive build. So exclude those explicitly.
+            '--exclude', 'pip', '--exclude', 'wheel',
+            # enable/disable noarchive
+            *debug_args,
+        ],
+        app_args=[
+            package,
+            '--output-file', out_frozen,
+        ]
+    )
+    # Read results
+    results_frozen = _read_results_file(out_frozen)
+
+    # Compare
+    assert results_unfrozen == results_frozen