Repository: pyinstaller/pyinstaller

Issue:
pkgutil.iter_modules is broken on MacOS
## Description of the issue

I am trying to build an application on MacOS that uses pkguilt.iter_modules to dynamically find and load modules however it is not finding any modules.
The modules are there and can be imported with importlib.import_module.
Note that I am using PyInstaller with FBS but I don't believe the latter changes anything that would effect this.

The following is an example project structure
```
my_package
    my_module_1
    my_module_2
    my_module_3
```

Here is the code that does not work
```py
import importlib
import pkgutil
m = importlib.import_module("my_package")
print("Found nested modules", list(pkgutil.iter_modules(m.__path__, m.__name__ + ".")))
```

The last line prints an empty list when built with PyInstaller.

I have done a bit of a dive into the code and I believe this is a rather complex mixup with paths and symbolic links.
https://github.com/pyinstaller/pyinstaller/blob/develop/PyInstaller/hooks/rthooks/pyi_rth_pkgutil.py

It looks like #6539 had a go at supporting symbolic links but it only resolves symbolic links up to the `_MEIPASS` directory.

On MacOS `_MEIPASS` is equal to `"/path/to/my_package.app/Contents/MacOS/"`

`my_package.__path__` is equal to `["/path/to/my_package.app/Contents/MacOS/my_package"]`

`/path/to/my_package.app/Contents/MacOS/my_package` is a symbolic link to `/path/to/my_package.app/Contents/Resources/my_package` when there are data files present in that package.

The `iter_modules` hook expands symbolic links in the `path` variable which converts `"/path/to/my_package.app/Contents/MacOS/my_package"` to `"/path/to/my_package.app/Contents/Resources/my_package"`

The following code then skips over the directory because it does not start with `SYS_PREFIX`

```py
                if not pkg_path.startswith(SYS_PREFIX):
                    # If the path does not start with sys._MEIPASS, it cannot be a bundled package.
                    continue
```

I don't know how this should be implemented but we need to resolve the expected path relative to the `_MEIPASS` directory and then expand symbolic links and only then skip if the package path does not start with the directory.


### Context information (for bug reports)

* Output of `pyinstaller --version`: ```5.13.0```
* Version of Python: Python 3.11.4 (main, Aug 22 2023, 11:47:24) [Clang 14.0.3 (clang-1403.0.22.14.1)] on darwin
* Platform: OS X
* How you installed Python: pyenv
* Did you also try this on another platform? Does it work there? This only effects MacOS as far as I am aware



Discussion:
That bug report I just linked is another project that doesn't use FBS so I don't think it effects it.
> /path/to/my_package.app/Contents/MacOS/my_package is a symbolic link to /path/to/my_package.app/Contents/Resources/my_package however the code does not resolve that.

Are you adding your packages as data files then? Your package shouldn't be there at all -- it should be embedded inside the executable.
The actual code is embedded in the executable but the `__path__` attribute still resolves to the `Contents/Resources` directory.

The import mechanism may still be resolving them via `Contents/MacOS` which redirects to `Contents/Resources`
I have just put together a minimal test example to debug this behaviour.
What I said earlier was wrong. `my_package.__path__` is actually equal to `["/path/to/my_package.app/Contents/MacOS/my_package"]`

Your `iter_modules` hook then resolves the symbolic links and expands it to `"/path/to/my_package.app/Contents/Resources/my_package"` like I said earlier.

The symbolic links in `MacOS` are only created when there are data files in that package so the issue only happens when data files are present in that package.

I will update the original post

Resolving patch:
diff --git a/PyInstaller/hooks/rthooks/pyi_rth_pkgutil.py b/PyInstaller/hooks/rthooks/pyi_rth_pkgutil.py
--- a/PyInstaller/hooks/rthooks/pyi_rth_pkgutil.py
+++ b/PyInstaller/hooks/rthooks/pyi_rth_pkgutil.py
@@ -47,39 +47,39 @@ def _pyi_pkgutil_iter_modules(path=None, prefix=''):
         if path is None:
             # Search for all top-level packages/modules. These will have no dots in their entry names.
             for entry in importer.toc:
-                if entry.count('.') != 0:
+                if "." in entry:
                     continue
                 is_pkg = importer.is_package(entry)
                 yield pkgutil.ModuleInfo(importer, prefix + entry, is_pkg)
         else:
-            # Declare SYS_PREFIX locally, to avoid clash with eponymous global symbol from pyi_rth_pkgutil hook.
-            #
             # Use os.path.realpath() to fully resolve any symbolic links in sys._MEIPASS, in order to avoid path
             # mis-matches when the given search paths also contain symbolic links and are already fully resolved.
             # See #6537 for an example of such a problem with onefile build on macOS, where the temporary directory
             # is placed under /var, which is actually a symbolic link to /private/var.
-            SYS_PREFIX = os.path.realpath(sys._MEIPASS) + os.path.sep
-            SYS_PREFIXLEN = len(SYS_PREFIX)
+            MEIPASS = os.path.realpath(sys._MEIPASS)
 
             for pkg_path in path:
                 # Fully resolve the given path, in case it contains symbolic links.
                 pkg_path = os.path.realpath(pkg_path)
-                if not pkg_path.startswith(SYS_PREFIX):
-                    # If the path does not start with sys._MEIPASS, it cannot be a bundled package.
-                    continue
-                # Construct package prefix from path...
-                pkg_prefix = pkg_path[SYS_PREFIXLEN:]
-                pkg_prefix = pkg_prefix.replace(os.path.sep, '.')
-                # ... and ensure it ends with a dot (so we can directly filter out the package itself).
-                if not pkg_prefix.endswith('.'):
-                    pkg_prefix += '.'
-                pkg_prefix_len = len(pkg_prefix)
+                # Ensure it ends with os.path.sep (so we can directly filter out the package itself).
+                if not pkg_path.endswith(os.path.sep):
+                    pkg_path += os.path.sep
 
                 for entry in importer.toc:
-                    if not entry.startswith(pkg_prefix):
+                    # Get the path to the module, and resolve symbolic links. This implicitly solves the problem with
+                    # macOS .app bundles and packages that contain data files, where package directory is fully
+                    # collected into Contents/Resources, and symbolic link to directory is created in sys._MEIPASS
+                    # (Contents/Frameworks). See #7884.
+                    module_path = os.path.realpath(os.path.join(MEIPASS, entry.replace(".", os.path.sep)))
+
+                    # If the module is not in the requested path, skip it.
+                    if not module_path.startswith(pkg_path):
                         continue
-                    name = entry[pkg_prefix_len:]
-                    if name.count('.') != 0:
+
+                    name = module_path[len(pkg_path):]
+
+                    if os.path.sep in name:
+                        # Not a direct child
                         continue
                     is_pkg = importer.is_package(entry)
                     yield pkgutil.ModuleInfo(importer, prefix + name, is_pkg)


Test patch:
diff --git a/tests/functional/modules/pyi_pkgutil_itermodules/hooks/hook-mypackage.py b/tests/functional/modules/pyi_pkgutil_itermodules/hooks/hook-mypackage.py
new file mode 100644
--- /dev/null
+++ b/tests/functional/modules/pyi_pkgutil_itermodules/hooks/hook-mypackage.py
@@ -0,0 +1,15 @@
+#-----------------------------------------------------------------------------
+# Copyright (c) 2020-2023, PyInstaller Development Team.
+#
+# Distributed under the terms of the GNU General Public License (version 2
+# or later) with exception for distributing the bootloader.
+#
+# The full license is in the file COPYING.txt, distributed with this software.
+#
+# SPDX-License-Identifier: (GPL-2.0-or-later WITH Bootloader-exception)
+#-----------------------------------------------------------------------------
+
+from PyInstaller.utils.hooks import collect_data_files, collect_submodules
+
+hiddenimports = collect_submodules('mypackage')
+datas = collect_data_files('mypackage')
diff --git a/tests/functional/modules/pyi_pkgutil_itermodules/package/mypackage/__init__.py b/tests/functional/modules/pyi_pkgutil_itermodules/package/mypackage/__init__.py
new file mode 100644
--- /dev/null
+++ b/tests/functional/modules/pyi_pkgutil_itermodules/package/mypackage/__init__.py
@@ -0,0 +1 @@
+#
diff --git a/tests/functional/modules/pyi_pkgutil_itermodules/package/mypackage/datafile.txt b/tests/functional/modules/pyi_pkgutil_itermodules/package/mypackage/datafile.txt
new file mode 100644
--- /dev/null
+++ b/tests/functional/modules/pyi_pkgutil_itermodules/package/mypackage/datafile.txt
@@ -0,0 +1 @@
+An example data file.
diff --git a/tests/functional/modules/pyi_pkgutil_itermodules/package/mypackage/submodule_a.py b/tests/functional/modules/pyi_pkgutil_itermodules/package/mypackage/submodule_a.py
new file mode 100644
--- /dev/null
+++ b/tests/functional/modules/pyi_pkgutil_itermodules/package/mypackage/submodule_a.py
@@ -0,0 +1 @@
+#
diff --git a/tests/functional/modules/pyi_pkgutil_itermodules/package/mypackage/submodule_b.py b/tests/functional/modules/pyi_pkgutil_itermodules/package/mypackage/submodule_b.py
new file mode 100644
--- /dev/null
+++ b/tests/functional/modules/pyi_pkgutil_itermodules/package/mypackage/submodule_b.py
@@ -0,0 +1 @@
+#
diff --git a/tests/functional/modules/pyi_pkgutil_itermodules/package/mypackage/submodule_c.py b/tests/functional/modules/pyi_pkgutil_itermodules/package/mypackage/submodule_c.py
new file mode 100644
--- /dev/null
+++ b/tests/functional/modules/pyi_pkgutil_itermodules/package/mypackage/submodule_c.py
@@ -0,0 +1 @@
+#
diff --git a/tests/functional/test_pkgutil.py b/tests/functional/test_pkgutil.py
--- a/tests/functional/test_pkgutil.py
+++ b/tests/functional/test_pkgutil.py
@@ -28,6 +28,9 @@
 from PyInstaller.compat import exec_python_rc
 from PyInstaller.utils.tests import importable
 
+# Directory with testing modules used in some tests.
+_MODULES_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'modules')
+
 
 # Read the output file produced by test script. Each line consists of two elements separated by semi-colon:
 # name;ispackage
@@ -100,3 +103,71 @@ def test_pkgutil_iter_modules_resolve_pkg_path(script_dir, tmpdir, pyi_builder):
         pytest.skip('The test is applicable only to onefile mode.')
     # A single combination (altgraph package, archive mode) is enough to check for proper symlink handling.
     test_pkgutil_iter_modules('json', script_dir, tmpdir, pyi_builder, archive=True, resolve_pkg_path=True)
+
+
+# Additional test for macOS .app bundles and packages that contain data files. See #7884. In generated .app bundles,
+# _MEIPASS points to `Contents/Frameworks`, while the data files are collected into `Contents/Resources` directory. If
+# a package contains only data files, the whole package directory is collected into `Contents/Resources`, and a symbolic
+# link to package's directory is made in `Contents/Frameworks`. Our `pkgutil.iter_modules` implementation needs to
+# account for this when validating the package path prefix; i.e., that attempting to resolve
+# `Contents/Frameworks/mypackage` will result in `Contents/Resource/mypackage` due to symbolic link, and thus the prefix
+# will not directly match _MEIPASS anymore.
+#
+# This issue affects packages with only data files; if the package has no data or binary files, then the package
+# directory does not exist on filesystem and the resolution attempt leaves it unchanged. If the package contains both
+# data and binary files, the directory is created in both Contents/Frameworks and Contents/Resources, and the contents
+# are cross-linked between them on file level.
+@pytest.mark.darwin
+def test_pkgutil_iter_modules_macos_app_bundle(script_dir, tmpdir, pyi_builder, monkeypatch):
+    if pyi_builder._mode != 'onedir':
+        pytest.skip('The test is applicable only to onedir mode.')
+
+    pathex = os.path.join(_MODULES_DIR, 'pyi_pkgutil_itermodules', 'package')
+    hooks_dir = os.path.join(_MODULES_DIR, 'pyi_pkgutil_itermodules', 'hooks')
+    package = 'mypackage'
+
+    # Full path to test script
+    test_script = 'pyi_pkgutil_iter_modules.py'
+    test_script = os.path.join(script_dir, test_script)
+
+    # Run unfrozen test script
+    env = os.environ.copy()
+    if 'PYTHONPATH' in env:
+        pathex = os.pathsep.join([pathex, env['PYTHONPATH']])
+    env['PYTHONPATH'] = pathex
+    out_unfrozen = os.path.join(tmpdir, 'output-unfrozen.txt')
+    rc = exec_python_rc(test_script, package, '--output-file', out_unfrozen, env=env)
+    assert rc == 0
+    # Read results
+    results_unfrozen = _read_results_file(out_unfrozen)
+
+    # Freeze the test program
+    # This also runs both executables (POSIX build and .app bundle) with same arguments, so we have no way of separating
+    # the output file. Therefore, we will manually re-run the executables ourselves.
+    pyi_builder.test_script(
+        test_script,
+        pyi_args=[
+            '--paths', pathex,
+            '--hiddenimport', package,
+            '--additional-hooks-dir', hooks_dir,
+            '--windowed',  # enable .app bundle
+        ],
+        app_args=[package],
+    )  # yapf: disable
+
+    # Run each executable and verify its output
+    executables = pyi_builder._find_executables('pyi_pkgutil_iter_modules')
+    assert executables
+    for idx, exe in enumerate(executables):
+        out_frozen = os.path.join(tmpdir, f"output-frozen-{idx}.txt")
+        rc = pyi_builder._run_executable(
+            exe,
+            args=[package, '--output-file', out_frozen],
+            run_from_path=False,
+            runtime=None,
+        )
+        assert rc == 0
+        results_frozen = _read_results_file(out_frozen)
+        print("RESULTS", results_frozen, "\n\n")
+
+        assert results_unfrozen == results_frozen