Repository: pyinstaller/pyinstaller

Issue:
`utils.hooks.collect_submodules()` does not apply the filter before doing a test import, generating errors for modules that won't be collected
<!--
Welcome to the PyInstaller issue tracker! Before creating an issue, please heed the following:

1. This tracker should only be used to report bugs and request features / enhancements to PyInstaller
    - For questions and general support, use the mailing list, see
	  <http://www.pyinstaller.org/support.html#mailing-list>
2. Use the search function before creating a new issue. Duplicates will be closed and directed to
   the original discussion.
3. When making a bug report, make sure you provide all required information. The easier it is for
   maintainers to reproduce, the faster it'll be fixed.
-->

<!-- +++ ONLY TEXT +++ DO NOT POST IMAGES +++ -->

## Description of the issue

[`utils.hooks.collect_submodules()`](https://github.com/pyinstaller/pyinstaller/blob/3b5d233d02472ad9d589a798d19ad3d3df645223/PyInstaller/utils/hooks/__init__.py#L595) accepts an optional `filter` argument that defines a callable to use to filter which modules should be included in the collected modules. `collect_submodules` first tries to discover all of the submodules of the given `package` and tries to test that the submodule can be imported. This occurs inside `_collect_submodules()` (line 681). 

The problem is that the filter given to `collect_submodules` is not applied to the module name prior to testing the module import. Therefore, if the filter is important for knowing which modules should be ignored (possibly due to known import restrictions), then nuisance errors will occur. The workaround is that `on_error` can be set to `'ignore'`, but obviously that prevents *actually useful* errors from being reported too. If `on_error='raise'`, then `collect_submodules` cannot even be used at all!

### Extra details

Another weird thing is that `_collect_submodules` leaves the module name in the list of discovered modules when the module cannot be imported. It seems a bit strange that it does this (why is it even testing the import in the first place then?). I guess the import test prevents the function from searching that module's submodules (but if this module is un-importable *and is kept*, why would you ignore its un-importable submodules?). I feel that if the submodule cannot be imported, then it should be removed from the list of collected modules (but continuing to generate any errors as per `on_error`).

I attempted to hack a quick fix together but was unsuccessful. I'll describe what I encountered here to prevent headaches for anyone else trying to fix this:
>I tried to pass the `filter` argument to `_collect_submodules`, but that function runs in an isolated process due to the `@isolated.decorate` decorator. That causes a problem because when the function call is sent to the isolated process, the `marshal` package is unable to dump the `filter` argument (due to being a callable). To try to work around this, I tried out the `pickle` package as a replacement to `marshal` but got a similar error with the `marshal` package.

I don't really have any suggestions on how to fix this. `_collect_submodules` needs to run the filter on the submodule names it discovers but needs to run isolated, so the filter cannot be used in the function. And `collect_submodules` has access to the filter, but it doesn't know all of the module names yet, since `_collect_submodules` discovers submodules by recursing into submodules if the import worked.

Perhaps the error logging part needs to be refactored. If the error logging part were to happen after `collect_submodules` could filter out the ignored files (by running the same import tests from `_collect_submodules` on the *filtered* module list), then nuisance errors wouldn't be generated, and `on_error='raise'` could be used normally.

### Context information (for bug reports)

* Output of `pyinstaller --version`: ```5.1```
* Version of Python: ```3.9.5```
* Platform: Windows 10
* How you installed Python: python.org/downloads
* Did you also try this on another platform? No

### Make sure [everything is packaged correctly](https://github.com/pyinstaller/pyinstaller/wiki/How-to-Report-Bugs#make-sure-everything-is-packaged-correctly)

**N/A**

  * [x] Repackage you **(SIC)** application in **verbose/debug mode**...


### A minimal example program which shows the error

From https://github.com/pyinstaller/pyinstaller-hooks-contrib/blob/master/src/_pyinstaller_hooks_contrib/hooks/stdhooks/hook-pylint.py in an environment that has `pylint` installed but `pytest` is not installed.
```
# Add imports from dynamically loaded modules excluding tests and testutils
hiddenimports = collect_submodules('pylint',
                                   lambda name: (not is_module_or_submodule(name, 'pylint.test')) and
                                   (not name == 'testutils'))
```

Note: `hook-pylint.py` seems to have a bug in `not name == 'testutils'`. I think it should be `not is_module_or_submodule(name, 'pylint.testutils')`. When I change this in my local `hook-pylint.py`, it doesn't fix the issue, so it isn't the source of this bug. Also, I don't know what `pylint.test` is. I don't think that `pylint` has any submodules named `test` anymore.

### Stacktrace / full error message


```
    15569 INFO: Loading module hook 'hook-pylint.py' from '[redacted]\\venv\\lib\\site-packages\\_pyinstaller_hooks_contrib\\hooks\\stdhooks'...
    739 WARNING: Failed to collect submodules for 'pylint.testutils' because importing 'pylint.testutils' raised: ModuleNotFoundError: No module named '_pytest'
```


Discussion:
I agree that passing the filter into the isolated subprocess should be the answer but as you've already discovered - having a function parameter will not be fun. The only way I know to serialise functions is to marshal the code object like we do [here](https://github.com/pyinstaller/pyinstaller/blob/3b5d233d02472ad9d589a798d19ad3d3df645223/PyInstaller/isolated/_parent.py#L154-L156) and reconstruct it like we do [here](https://github.com/pyinstaller/pyinstaller/blob/develop/PyInstaller/isolated/_child.py#L58-L60) and hope that no-one gets too confused by the global namespace the function runs in being empty or tries to use a custom callable or builtin function as the filter.
Although one very unhelpful and opinionated solution is to say that packages ([including PyInstaller](https://github.com/pyinstaller/pyinstaller/blob/develop/PyInstaller/utils/tests.py)) shouldn't contain their own test suites or test helpers. Then we wouldn't need to exclude bits of packages in the first place.
I worked out an implementation for the solution to delay the error reporting. It's probably not the best way to solve it, and it's pretty inefficient comparatively.

[utils/hooks/\_\_init\_\_.py](https://github.com/pyinstaller/pyinstaller/blob/3b5d233d02472ad9d589a798d19ad3d3df645223/PyInstaller/utils/hooks/__init__.py#L595):
```python
def collect_submodules(package: str, filter: Callable[[str], bool] = lambda name: True, on_error="warn once"):
    # Accept only strings as packages.
    if not isinstance(package, compat.string_types):
        raise TypeError('package must be a str')
    if on_error not in ("ignore", "warn once", "warn", "raise"):
        raise ValueError(
            f"Invalid on-error action '{on_error}': Must be one of ('ignore', 'warn once', 'warn', 'raise')"
        )

    logger.debug('Collecting submodules for %s' % package)
    # Skip a module which is not a package.
    if not is_package(package):
        logger.debug('collect_submodules - Module %s is not a package.' % package)
        return []

    # Determine the filesystem path(s) to the specified package.
    pkg_dirs = get_all_package_paths(package)
    modules = []
    for pkg_dir in pkg_dirs:
        modules += _collect_submodules(pkg_dir, package)

    # Apply the user defined filter function. Filter out duplicates at the same time.
    modules = [i for i in set(modules) if filter(i)]

    # Find any errors in the collected modules, and handle appropriately.
    _find_errors_in_collected_submodules(modules, package, on_error)

    logger.debug("collect_submodules - Found submodules: %s", modules)
    return modules


@isolated.decorate
def _collect_submodules(pkg_dir, package):
    import sys
    import pkgutil
    from collections import deque

    # ``pkgutil.walk_packages`` doesn't walk subpackages of zipped files per https://bugs.python.org/issue14209. This is
    # a workaround.
    seen = set()
    todo = deque([([pkg_dir], package + ".")])
    modules = [package]

    while todo:
        path, prefix = todo.pop()
        for importer, name, ispkg in pkgutil.iter_modules(path, prefix):
            if not name.startswith(prefix):
                name = prefix + name
            modules.append(name)

            if not ispkg:
                # Only packages can have submodules.
                continue

            try:
                __import__(name)
            except Exception as ex:
                # Don't attempt to recurse to submodules if this module didn't even make it into sys.modules.
                if name not in sys.modules:
                    continue
            path = getattr(sys.modules[name], '__path__', None) or []

            # Don't traverse path items we've seen before.
            path = [p for p in path if p not in seen]
            seen.update(path)
            todo.append((path, name + '.'))

    return modules


@isolated.decorate
def _find_errors_in_collected_submodules(modules, package, on_error):
    from importlib.util import find_spec
    from traceback import format_exception_only

    for name in modules:
        if name == package:
            # Only find errors for submodules of the package.
            continue

        ispkg = find_spec(name).submodule_search_locations is not None
        if not ispkg:
            continue

        try:
            __import__(name)
        except Exception as ex:
            # Catch all errors then either raise, warn or ignore them as determined by the *on_error* parameter.
            if on_error in ("warn", "warn once"):
                from PyInstaller.log import logger
                ex = "".join(format_exception_only(type(ex), ex)).strip()
                logger.warning(f"Failed to collect submodules for '{name}' because importing '{name}' raised: {ex}")
                if on_error == "warn once":
                    on_error = "ignore"
            elif on_error == "raise":
                raise ImportError(f"Unable to load submodule '{name}'.") from ex
```
Hmm, surely a delayed error reporting could be implemented by collecting error messages during the initial walk and returning them from subprocess along with the list of modules? Then dropping the ones corresponding to filtered modules, and reporting/raising the rest?

But I think it would make sense to reorganize the code so that filter actually prevents the attempt to walk into an excluded subpackage. Because post-processing does not prevent us from walking into an optional (i.e., not imported by the parent's `__init__.py`) subpackage that crashes the python (e.g., by trying to load a bogus extension). If filter prevented walking into such sub-package, such (known) faulty sub-packages could be avoided altogether. Currently, we have no mechanism for that.

I suppose the code could be broken up so that the parent process requests the isolated child to perform a scan of current package and report results for that recursion level immediately (e.g., (name, path, is_pkg) tuples). Then based on the filter, it decides the next sub-package to recurse into, and repeat the process. At the very naive level, this could be done using `@isolated.decorate` decorator and spawning subprocess for each recursion step, but we could probably also come up with some sort of state machine living in the remote process, and communication implemented via lower-level `isolated.Python.call` (so we avoid needlessly repeatedly importing the higher-level packages). We might need to adjust the `isolated` code to preserve the globals between subsequent calls in the session, though...

Resolving patch:
diff --git a/PyInstaller/utils/hooks/__init__.py b/PyInstaller/utils/hooks/__init__.py
--- a/PyInstaller/utils/hooks/__init__.py
+++ b/PyInstaller/utils/hooks/__init__.py
@@ -14,6 +14,7 @@
 import sys
 import textwrap
 from pathlib import Path
+from collections import deque
 from typing import Callable, Tuple
 
 import pkg_resources
@@ -634,72 +635,92 @@ def collect_submodules(package: str, filter: Callable[[str], bool] = lambda name
             f"Invalid on-error action '{on_error}': Must be one of ('ignore', 'warn once', 'warn', 'raise')"
         )
 
-    logger.debug('Collecting submodules for %s' % package)
+    logger.debug('Collecting submodules for %s', package)
+
     # Skip a module which is not a package.
     if not is_package(package):
-        logger.debug('collect_submodules - Module %s is not a package.' % package)
+        logger.debug('collect_submodules - %s is not a package.', package)
+        # If module is importable, return its name in the list, in order to keep behavior consistent with the
+        # one we have for packages (i.e., we include the package in the list of returned names)
+        if can_import_module(package):
+            return [package]
         return []
 
     # Determine the filesystem path(s) to the specified package.
-    pkg_dirs = get_all_package_paths(package)
-    modules = []
-    for pkg_dir in pkg_dirs:
-        modules += _collect_submodules(pkg_dir, package, on_error)
+    package_submodules = []
 
-    # Apply the user defined filter function. Filter out duplicates at the same time.
-    modules = [i for i in set(modules) if filter(i)]
+    todo = deque()
+    todo.append(package)
 
-    logger.debug("collect_submodules - Found submodules: %s", modules)
-    return modules
+    with isolated.Python() as isolated_python:
+        while todo:
+            # Scan the given (sub)package
+            name = todo.pop()
+            modules, subpackages, on_error = isolated_python.call(_collect_submodules, name, on_error)
 
+            # Add modules to the list of all submodules
+            package_submodules += [module for module in modules if filter(module)]
 
-@isolated.decorate
-def _collect_submodules(pkg_dir, package, on_error):
+            # Add sub-packages to deque for subsequent recursion
+            for subpackage_name in subpackages:
+                if filter(subpackage_name):
+                    todo.append(subpackage_name)
+
+    package_submodules = sorted(package_submodules)
+
+    logger.debug("collect_submodules - found submodules: %s", package_submodules)
+    return package_submodules
+
+
+# This function is called in an isolated sub-process via `isolated.Python.call`.
+def _collect_submodules(name, on_error):
     import sys
     import pkgutil
     from traceback import format_exception_only
-    from collections import deque
 
-    # ``pkgutil.walk_packages`` doesn't walk subpackages of zipped files per https://bugs.python.org/issue14209. This is
-    # a workaround.
-    seen = set()
-    todo = deque([([pkg_dir], package + ".")])
-    modules = [package]
+    from PyInstaller.utils.hooks import logger
 
-    while todo:
-        path, prefix = todo.pop()
-        for importer, name, ispkg in pkgutil.iter_modules(path, prefix):
-            if not name.startswith(prefix):
-                name = prefix + name
+    logger.debug("collect_submodules - scanning (sub)package %s", name)
+
+    modules = []
+    subpackages = []
+
+    # Resolve package location(s)
+    try:
+        __import__(name)
+    except Exception as ex:
+        # Catch all errors and either raise, warn, or ignore them as determined by the *on_error* parameter.
+        if on_error in ("warn", "warn once"):
+            from PyInstaller.log import logger
+            ex = "".join(format_exception_only(type(ex), ex)).strip()
+            logger.warning(f"Failed to collect submodules for '{name}' because importing '{name}' raised: {ex}")
+            if on_error == "warn once":
+                on_error = "ignore"
+            return modules, subpackages, on_error
+        elif on_error == "raise":
+            raise ImportError(f"Unable to load subpackage '{name}'.") from ex
+
+    # Do not attempt to recurse into package if it did not make it into sys.modules.
+    if name not in sys.modules:
+        return modules, subpackages, on_error
+
+    # Or if it does not have __path__ attribute.
+    paths = getattr(sys.modules[name], '__path__', None) or []
+    if not paths:
+        return modules, subpackages, on_error
+
+    # Package was successfully imported - include it in the list of modules.
+    modules.append(name)
+
+    # Iterate package contents
+    logger.debug("collect_submodules - scanning (sub)package %s in location(s): %s", name, paths)
+    for importer, name, ispkg in pkgutil.iter_modules(paths, name + '.'):
+        if not ispkg:
             modules.append(name)
+        else:
+            subpackages.append(name)
 
-            if not ispkg:
-                # Only packages can have submodules.
-                continue
-
-            try:
-                __import__(name)
-            except Exception as ex:
-                # Catch all errors then either raise, warn or ignore them as determined by the *on_error* parameter.
-                if on_error in ("warn", "warn once"):
-                    from PyInstaller.log import logger
-                    ex = "".join(format_exception_only(type(ex), ex)).strip()
-                    logger.warning(f"Failed to collect submodules for '{name}' because importing '{name}' raised: {ex}")
-                    if on_error == "warn once":
-                        on_error = "ignore"
-                elif on_error == "raise":
-                    raise ImportError(f"Unable to load submodule '{name}'.") from ex
-                # Don't attempt to recurse to submodules if this module didn't even make it into sys.modules.
-                if name not in sys.modules:
-                    continue
-            path = getattr(sys.modules[name], '__path__', None) or []
-
-            # Don't traverse path items we've seen before.
-            path = [p for p in path if p not in seen]
-            seen.update(path)
-            todo.append((path, name + '.'))
-
-    return modules
+    return modules, subpackages, on_error
 
 
 def is_module_or_submodule(name, mod_or_submod):


Test patch:
diff --git a/tests/unit/test_hookutils.py b/tests/unit/test_hookutils.py
--- a/tests/unit/test_hookutils.py
+++ b/tests/unit/test_hookutils.py
@@ -118,10 +118,11 @@ def mod_list(monkeypatch):
 
 class TestCollectSubmodules(object):
     # A message should be emitted if a module, not a package, was passed.
+    # The module's name should be in the returned list, nevetheless.
     def test_collect_submod_module(self, caplog):
         with caplog.at_level(logging.DEBUG, logger='PyInstaller.utils.hooks'):
-            assert collect_submodules('os') == []
-            assert "Module os is not a package." in caplog.records[-1].msg
+            assert collect_submodules('os') == ['os']
+            assert "collect_submodules - os is not a package." in caplog.records[-1].getMessage()
 
     # A TypeError should be raised if given something other than a str.
     def test_not_a_string(self):
@@ -137,14 +138,15 @@ def test_collect_submod_pyextension(self, mod_list):
         assert TEST_MOD + '.pyextension' in mod_list
 
     # Check that all packages get included
+    # NOTE: the new behavior (see #6846 and #6850) is that un-importable subpackages are not included.
     def test_collect_submod_all_included(self, mod_list):
         mod_list.sort()
         assert mod_list == [
             TEST_MOD,
             # Python extensions end with '.pyd' on Windows and with  '.so' on Linux, Mac OS, and other OSes.
             TEST_MOD + '.pyextension',
-            TEST_MOD + '.raises_error_on_import_1',
-            TEST_MOD + '.raises_error_on_import_2',
+            #TEST_MOD + '.raises_error_on_import_1',
+            #TEST_MOD + '.raises_error_on_import_2',
             TEST_MOD + '.subpkg',
             TEST_MOD + '.subpkg.twelve',
             TEST_MOD + '.two'