Repository: pyinstaller/pyinstaller

Issue:
Flow of `excludedimports` option
## Description of the issue
It seems that the `--excludedimports` option for hooks doesn't match what [the docs](https://pyinstaller.org/en/stable/hooks.html#hook-global-variables) say about it: _If an excluded name is explicitly imported in the source file or some other module, it will be kept_. Below, in the given example, [the SQLAlchemy hook](https://github.com/pyinstaller/pyinstaller/blob/develop/PyInstaller/hooks/hook-sqlalchemy.py#L22) excludes import of `sqlalchemy.testing`, but its explicit import in the source code leads to a crashing app. 
I have tested and found out that such behaviour starts with version `5.10.0`. Versions before it can build and run the example successfully. As can be seen from [the Changelog](https://pyinstaller.org/en/stable/CHANGES.html#id117), the option logic was changed there, specifically in #7495.

### Context information (for bug reports)

* Output of `pyinstaller --version`: `6.14.2` (Taken from the `develop` branch, revision `6966b11767d1074fa3acd7fc7a66f4336d8aa54c`)
* Version of Python: `3.10.11`
* Platform: Windows 11 Pro 24H2
* How you installed Python: Windows Store
* `SQLAlchemy` version: `1.4.54` (Also tested with `2.0.35` version)

### Make sure [everything is packaged correctly](https://github.com/pyinstaller/pyinstaller/wiki/How-to-Report-Bugs#make-sure-everything-is-packaged-correctly)

  * [x] start with a clean installation
  * [x] use the latest development version
  * [x] Run your frozen program **from a command window (shell)** — instead of double-clicking on it
  * [x] Package your program in **--onedir mode**
  * [x] Package **without UPX**, say: use the option `--noupx` or set `upx=False` in your .spec-file
  * [x] Repackage you application in **verbose/debug mode**. For this, pass the option `--debug` to `pyi-makespec` or `pyinstaller` or use `EXE(..., debug=1, ...)` in your .spec file.


### A minimal example program which shows the error

`test.py`
```python3
import sqlalchemy.testing
```

### Stacktrace / full error message

```log
Traceback (most recent call last):
  File "test.py", line 1, in <module>
    import sqlalchemy.testing
  File "PyInstaller\loader\pyimod02_importers.py", line 457, in exec_module
  File "sqlalchemy\testing\__init__.py", line 9, in <module>
ImportError: cannot import name 'config' from partially initialized module 'sqlalchemy.testing' (most likely due to a circular import) (C:\Users\Михаил\Desktop\dist\test\_internal\sqlalchemy\testing\__init__.py)
[PYI-15496:ERROR] Failed to execute script 'test' due to unhandled exception!
```

<details><summary>Applcation log with bootloader debug</summary>

```log
C:\Users\Михаил\Desktop>dist\test\test.exe
[PYI-10232:DEBUG] PyInstaller Bootloader 6.x
[PYI-10232:DEBUG] LOADER: argv[0]: dist\test\test.exe
[PYI-10232:DEBUG] LOADER: executable file: C:\Users\??????\Desktop\dist\test\test.exe
[PYI-10232:DEBUG] LOADER: trying to load executable-embedded archive...
[PYI-10232:DEBUG] LOADER: attempting to open archive C:\Users\??????\Desktop\dist\test\test.exe
[PYI-10232:DEBUG] LOADER: cookie found at offset 0x4AD9E9
[PYI-10232:DEBUG] LOADER: archive file: C:\Users\??????\Desktop\dist\test\test.exe
[PYI-10232:DEBUG] LOADER: application has onedir semantics...
[PYI-10232:DEBUG] LOADER: _PYI_ARCHIVE_FILE not defined...
[PYI-10232:DEBUG] LOADER: parent process level = -2
[PYI-10232:DEBUG] LOADER: process level = 1
[PYI-10232:DEBUG] LOADER: application's top-level directory: C:\Users\??????\Desktop\dist\test\_internal
[PYI-10232:DEBUG] LOADER: calling SetDllDirectoryW: C:\Users\??????\Desktop\dist\test\_internal
[PYI-10232:DEBUG] LOADER: splash screen is unavailable.
[PYI-10232:DEBUG] DYLIB: loading Python DLL: C:\Users\??????\Desktop\dist\test\_internal\python310.dll
[PYI-10232:DEBUG] DYLIB: loaded Python shared library.
[PYI-10232:DEBUG] DYLIB: imported symbols from Python shared library.
[PYI-10232:DEBUG] LOADER: pre-initializing embedded python interpreter...
[PYI-10232:DEBUG] LOADER: creating PyConfig structure...
[PYI-10232:DEBUG] LOADER: initializing interpreter configuration...
[PYI-10232:DEBUG] LOADER: setting program name...
[PYI-10232:DEBUG] LOADER: setting python home path...
[PYI-10232:DEBUG] LOADER: setting module search paths...
[PYI-10232:DEBUG] LOADER: setting sys.argv...
[PYI-10232:DEBUG] LOADER: applying run-time options...
[PYI-10232:DEBUG] LOADER: starting embedded python interpreter...
[PYI-10232:DEBUG] LOADER: setting sys._MEIPASS
[PYI-10232:DEBUG] LOADER: importing modules from PKG/CArchive
[PYI-10232:DEBUG] LOADER: extracted struct
[PYI-10232:DEBUG] LOADER: running unmarshalled code object for module struct...
[PYI-10232:DEBUG] LOADER: extracted pyimod01_archive
[PYI-10232:DEBUG] LOADER: running unmarshalled code object for module pyimod01_archive...
[PYI-10232:DEBUG] LOADER: extracted pyimod02_importers
[PYI-10232:DEBUG] LOADER: running unmarshalled code object for module pyimod02_importers...
[PYI-10232:DEBUG] LOADER: extracted pyimod03_ctypes
[PYI-10232:DEBUG] LOADER: running unmarshalled code object for module pyimod03_ctypes...
[PYI-10232:DEBUG] LOADER: extracted pyimod04_pywin32
[PYI-10232:DEBUG] LOADER: running unmarshalled code object for module pyimod04_pywin32...
[PYI-10232:DEBUG] LOADER: looking for PYZ archive TOC entry...
[PYI-10232:DEBUG] LOADER: path to PYZ archive stored into sys._pyinstaller_pyz...
[PYI-10232:DEBUG] LOADER: running pyiboot01_bootstrap.py
[PYI-10232:DEBUG] LOADER: running pyi_rth_inspect.py
[PYI-10232:DEBUG] LOADER: running pyi_rth_setuptools.py
[PYI-10232:DEBUG] LOADER: running pyi_rth_pkgutil.py
[PYI-10232:DEBUG] LOADER: running pyi_rth_multiprocessing.py
[PYI-10232:DEBUG] LOADER: running pyi_rth_cryptography_openssl.py
[PYI-10232:DEBUG] LOADER: running test.py
Traceback (most recent call last):
  File "test.py", line 1, in <module>
    import sqlalchemy.testing
  File "PyInstaller\loader\pyimod02_importers.py", line 457, in exec_module
  File "sqlalchemy\testing\__init__.py", line 9, in <module>
ImportError: cannot import name 'config' from partially initialized module 'sqlalchemy.testing' (most likely due to a circular import) (C:\Users\Михаил\Desktop\dist\test\_internal\sqlalchemy\testing\__init__.py)
[PYI-10232:ERROR] Failed to execute script 'test' due to unhandled exception!
[PYI-10232:DEBUG] LOADER: ERROR.
[PYI-10232:DEBUG] LOADER: manually flushing stdout and stderr...
[PYI-10232:DEBUG] LOADER: cleaning up Python interpreter...
[PYI-10232:DEBUG] DYLIB: unloading Python shared library...
[PYI-10232:DEBUG] DYLIB: unloaded Python shared library.
[PYI-10232:DEBUG] LOADER: end of process reached!
```
</details> 

Full `PyInstaller` debug build log - https://gist.github.com/mberdyshev/fefb0f7720bcb6df171bfceda04d248b
Especially look at steps **30091-30111** and **35492**!


Discussion:
Hmm, yeah, that's a problem with top-level module hooks excluding subpackages - even if those subpackages are collected due to a another reference / import chain, the submodules from such subpackages are not collected, because they are still suppressed by the exclusion of the subpackage (which is applied recursively). 

We will probably need to add support for ignoring exclusion of subpackages' modules when subpackage gets collected in spite of exclusion from a hook for its parent package. 

In the meantime, you will probably need to work around this by adding `--collect-submodules sqlalchemy.testing` to your PyInstaller command to force collection of all submodules from `sqlalchemy.testing`.

(I imagine the reason why it worked before #7495 is that relative imports from within the subpackage slipped through the exclusion, which was a bug in itself, even if it ended up cancelling out this issue).




Patch:
diff --git a/PyInstaller/depend/analysis.py b/PyInstaller/depend/analysis.py
index 3ca51e14ae8..a23cdae011b 100644
--- a/PyInstaller/depend/analysis.py
+++ b/PyInstaller/depend/analysis.py
@@ -397,16 +397,16 @@ def _safe_import_hook(
                         # Module in a package; base name must be the parent package name!
                         base_module_name = '.'.join(source_module.identifier.split('.')[:-1])
 
-                    if target_module_partname:
-                        base_module_name += '.' + target_module_partname
-
                     # Adjust the base module name based on level
                     if level > 1:
                         base_module_name = '.'.join(base_module_name.split('.')[:-(level - 1)])
+
+                    if target_module_partname:
+                        base_module_name += '.' + target_module_partname
                 else:
                     base_module_name = target_module_partname
 
-                def _exclude_module(module_name, excluded_imports):
+                def _exclude_module(module_name, excluded_imports, referrer_name):
                     """
                     Helper for checking whether given module should be excluded.
                     Returns the name of exclusion rule if module should be excluded, None otherwise.
@@ -416,13 +416,32 @@ def _exclude_module(module_name, excluded_imports):
                         excluded_import_parts = excluded_import.split('.')
                         match = module_name_parts[:len(excluded_import_parts)] == excluded_import_parts
                         if match:
+                            # Check if the referrer is (was!) subject to the same rule. Because if it was and was
+                            # analyzed anyway, some other import chain must have overrode the exclusion, and we should
+                            # waive it here. A package hook might exclude a part (a subpackage) of the said package to
+                            # prevent its collection when there are no external references; but when they are (for
+                            # example, user explicitly imports the said subpackage in their program), we must let the
+                            # subpackage import its submodules.
+                            referrer_name_parts = referrer_name.split('.')
+                            referrer_match = referrer_name_parts[:len(excluded_import_parts)] == excluded_import_parts
+                            if referrer_match:
+                                logger.debug(
+                                    "Deactivating suppression rule %r for module %r because it also applies to the "
+                                    "referrer (%r)...", excluded_import, module_name, referrer_name
+                                )
+                                continue
+
                             return excluded_import
                     return None
 
                 # First, check if base module name is to be excluded.
                 # This covers both basic `import a` and `import a.b.c`, as well as `from d import e, f` where base
                 # module `d` is excluded.
-                excluded_import_rule = _exclude_module(base_module_name, excluded_imports)
+                excluded_import_rule = _exclude_module(
+                    base_module_name,
+                    excluded_imports,
+                    source_module.identifier,
+                )
                 if excluded_import_rule:
                     logger.debug(
                         "Suppressing import of %r from module %r due to excluded import %r specified in a hook for %r "
@@ -437,7 +456,11 @@ def _exclude_module(module_name, excluded_imports):
                     filtered_target_attr_names = []
                     for target_attr_name in target_attr_names:
                         submodule_name = base_module_name + '.' + target_attr_name
-                        excluded_import_rule = _exclude_module(submodule_name, excluded_imports)
+                        excluded_import_rule = _exclude_module(
+                            submodule_name,
+                            excluded_imports,
+                            source_module.identifier,
+                        )
                         if excluded_import_rule:
                             logger.debug(
                                 "Suppressing import of %r from module %r due to excluded import %r specified in a hook "
diff --git a/news/9193.bugfix.rst b/news/9193.bugfix.rst
new file mode 100644
index 00000000000..5e6b6cf0cc9
--- /dev/null
+++ b/news/9193.bugfix.rst
@@ -0,0 +1,6 @@
+Attempt to mitigate the issue with module exclusion when a top-level
+package hook excludes its own subpackage to prevent its collection
+in the absence of any external references; such exclusion rule would
+prevent collection of modules from such subpackage even when it is
+supposed to be collected due to an external reference (for example, an
+explicit import from the user's program).
diff --git a/news/9197.bugfix.rst b/news/9197.bugfix.rst
new file mode 100644
index 00000000000..cf54bf57a41
--- /dev/null
+++ b/news/9197.bugfix.rst
@@ -0,0 +1,3 @@
+Fix a bug in module exclusion part of analysis codepath that would cause
+certain types of relative imports to be misinterpreted and thus fail to
+exclude them.


Test patch:
diff --git a/tests/functional/modules/pyi_excluded_subpackage/hooks/hook-mypackage.py b/tests/functional/modules/pyi_excluded_subpackage/hooks/hook-mypackage.py
new file mode 100644
index 00000000000..5bfb9aa59a7
--- /dev/null
+++ b/tests/functional/modules/pyi_excluded_subpackage/hooks/hook-mypackage.py
@@ -0,0 +1,2 @@
+# Exclude the optional subpackage
+excludedimports = ['mypackage.optional']
diff --git a/tests/functional/modules/pyi_excluded_subpackage/modules/mypackage/__init__.py b/tests/functional/modules/pyi_excluded_subpackage/modules/mypackage/__init__.py
new file mode 100644
index 00000000000..efbab0b0d02
--- /dev/null
+++ b/tests/functional/modules/pyi_excluded_subpackage/modules/mypackage/__init__.py
@@ -0,0 +1,1 @@
+from . import mandatory  # noqa: F401
diff --git a/tests/functional/modules/pyi_excluded_subpackage/modules/mypackage/mandatory/__init__.py b/tests/functional/modules/pyi_excluded_subpackage/modules/mypackage/mandatory/__init__.py
new file mode 100644
index 00000000000..816a026fdea
--- /dev/null
+++ b/tests/functional/modules/pyi_excluded_subpackage/modules/mypackage/mandatory/__init__.py
@@ -0,0 +1,1 @@
+from .a import an_important_symbol  # noqa: F401
diff --git a/tests/functional/modules/pyi_excluded_subpackage/modules/mypackage/mandatory/a.py b/tests/functional/modules/pyi_excluded_subpackage/modules/mypackage/mandatory/a.py
new file mode 100644
index 00000000000..4d67c2bd585
--- /dev/null
+++ b/tests/functional/modules/pyi_excluded_subpackage/modules/mypackage/mandatory/a.py
@@ -0,0 +1,44 @@
+# This module contains an "unused" function (not called from anywhere in the package itself) that adds reference to the
+# optional subpackage by importing a function from it. This simulates scenario where the said optional subpackage might
+# be excluded by the top-level package hook; so that it is collected only if some external code (user's program or 3rd
+# party package) is importing it.
+
+
+def an_unused_function(ref_type):
+    # A whole lot of different imports: subpackage vs submodule in it, function from package or submodule (the package
+    # imports it from submodule), relative vs. absolute imports. The goal is to ensure that all of these are properly
+    # interpreted by the exclusion mechanism, and end up being excluded as necessary.
+    if ref_type == 0:
+        import mypackage.optional
+        return mypackage.optional.optional_function()
+    elif ref_type == 1:
+        from mypackage import optional
+        return optional.optional_function()
+    elif ref_type == 2:
+        from .. import optional
+        return optional.optional_function()
+    elif ref_type == 3:
+        import mypackage.optional.b
+        return mypackage.optional.b.optional_function()
+    elif ref_type == 4:
+        from mypackage.optional import b
+        return b.optional_function()
+    elif ref_type == 5:
+        from ..optional import b
+        return b.optional_function()
+    elif ref_type == 6:
+        from mypackage.optional import optional_function
+        return optional_function()
+    elif ref_type == 7:
+        from ..optional import optional_function
+        return optional_function()
+    elif ref_type == 8:
+        from mypackage.optional.b import optional_function
+        return optional_function()
+    elif ref_type == 9:
+        from ..optional.b import optional_function
+        return optional_function()
+
+
+# This symbol is imported by mypackage.mandatory, so we cannot get away by excluding this module.
+an_important_symbol = 42
diff --git a/tests/functional/modules/pyi_excluded_subpackage/modules/mypackage/optional/__init__.py b/tests/functional/modules/pyi_excluded_subpackage/modules/mypackage/optional/__init__.py
new file mode 100644
index 00000000000..8f40c4a50c4
--- /dev/null
+++ b/tests/functional/modules/pyi_excluded_subpackage/modules/mypackage/optional/__init__.py
@@ -0,0 +1,1 @@
+from .b import optional_function  # noqa: F401
diff --git a/tests/functional/modules/pyi_excluded_subpackage/modules/mypackage/optional/b.py b/tests/functional/modules/pyi_excluded_subpackage/modules/mypackage/optional/b.py
new file mode 100644
index 00000000000..e67f3bfa191
--- /dev/null
+++ b/tests/functional/modules/pyi_excluded_subpackage/modules/mypackage/optional/b.py
@@ -0,0 +1,2 @@
+def optional_function():
+    return 42
diff --git a/tests/functional/test_module_exclusion.py b/tests/functional/test_module_exclusion.py
index f3a6e57cd83..5e7f77d2491 100644
--- a/tests/functional/test_module_exclusion.py
+++ b/tests/functional/test_module_exclusion.py
@@ -13,6 +13,11 @@
 
 import pytest
 
+from PyInstaller.utils.tests import importorskip, requires
+
+# Run the tests here in onedir mode only - onefile offers no additional insights in this context.
+pytestmark = pytest.mark.parametrize('pyi_builder', ['onedir'], indirect=True)
+
 # Directory with testing modules used in some tests.
 _MODULES_DIR = pathlib.Path(__file__).parent / 'modules'
 
@@ -49,3 +54,61 @@ def test_module_exclusion(exclude_args, exclude_hooks, pyi_builder):
         pyi_args=pyi_args,
         run_from_path=True
     )
+
+
+# Tests for excluded subpackage from the top-level package hook. Ensure that such subpackage is excluded when the only
+# reference comes from within the corresponding top-level package (or one of its submodules/subpackages). However, if
+# the subpackage is referred from external source (e.g., user's program), then it should be collected and the exclusion
+# rule from the top-level package hook should not block collection of submodules from that subpackage.
+@pytest.mark.parametrize("with_reference", [False, True], ids=["noref", "ref"])
+def test_subpackage_exclusion(pyi_builder, with_reference):
+    pyi_args = [
+        '--paths',
+        str(_MODULES_DIR / 'pyi_excluded_subpackage' / 'modules'),
+        '--additional-hooks-dir',
+        str(_MODULES_DIR / 'pyi_excluded_subpackage' / 'hooks'),
+    ]
+
+    if with_reference:
+        # Explicit reference to excluded subpackage in user's program; we expect the subpackage to be fully collected.
+        source_code = (
+            """
+            import mypackage.optional  # is importable if its submodules are also collected.
+            assert mypackage.optional.optional_function() == 42
+            """
+        )
+    else:
+        # No reference to excluded subpackage other than in top-level package; we expect the subpackage to be excluded.
+        source_code = (
+            """
+            import importlib.util
+
+            import mypackage
+
+            # Ensure the optional subpackage is unavailable
+            optional_spec = importlib.util.find_spec('mypackage.optional')
+            assert optional_spec == None, "mypackage.optional is collected, but should not be!"
+            """
+        )
+
+    pyi_builder.test_source(
+        source_code,
+        pyi_args=pyi_args,
+    )
+
+
+# Our hook for sqlalchemy excludes sqlalchemy.testing. Make sure explicitly importing the said subpackage works.
+# See #9193.
+@importorskip('sqlalchemy')
+def test_subpackage_exclusion_sqlalchemy_testing(pyi_builder):
+    pyi_builder.test_source("""
+        import sqlalchemy.testing
+        """)
+
+
+# Our hook for numpy excludes numpy.f2py for numpy < 2.0. Make sure explicitly importing the said subpackage works.
+@requires('numpy < 2.0')
+def test_subpackage_exclusion_numpy_f2py(pyi_builder):
+    pyi_builder.test_source("""
+        import numpy.f2py
+        """)