Repository: pyinstaller/pyinstaller

Issue:
pyi_rth_pkgutil.py hook might not work under MacOS in onefile mode
## Description of the issue
I'm using `iter_modules` in our cross-platform OpenSource Project [Specter-Desktop](https://github.com/cryptoadvance/specter-desktop) ([spec-file](https://github.com/cryptoadvance/specter-desktop/blob/master/pyinstaller/specterd.spec)).

For that, the [pyi_rth_pkgutil.py-hook](https://github.com/pyinstaller/pyinstaller/blob/v4.8/PyInstaller/hooks/rthooks/pyi_rth_pkgutil.py) is quite useful and solved my issue on the linux-platform. However, i have an issue with the MacOS-build at runtime:
```
$  ./dist/specterd server --config=DevelopmentConfig
Initializing HWI...
[2022-01-24 11:49:07,827] INFO in server: Configuration: cryptoadvance.specter.config.DevelopmentConfig
[2022-01-24 11:49:07,828] INFO in specter_migrator: Initiated MigDataManager(/Users/thomasmuller/.specter_dev/migration_data.json events:3 execs:2 )
Traceback (most recent call last):
  File "specterd.py", line 4, in <module>
    entry_point()
  File "click/core.py", line 829, in __call__
  File "click/core.py", line 782, in main
  File "click/core.py", line 1259, in invoke
  File "click/core.py", line 1066, in invoke
  File "click/core.py", line 610, in invoke
  File "cryptoadvance/specter/cli/cli_server.py", line 114, in server
  File "cryptoadvance/specter/server.py", line 104, in init_app
  File "cryptoadvance/specter/util/specter_migrator.py", line 125, in execute_migrations
  File "cryptoadvance/specter/util/specter_migrator.py", line 107, in plan_migration
  File "cryptoadvance/specter/util/specter_migrator.py", line 161, in get_migration_classes
  File "PyInstaller/hooks/rthooks/pyi_rth_pkgutil.py", line 59, in _pyi_pkgutil_iter_modules
AssertionError
[20771] Failed to execute script 'specterd' due to unhandled exception!
(.env) thomasmuller @ Thomass-Air.fritz.box ➜  pyinstaller git:(service_swan) ✗
```

So the relevant code is this:
```
        SYS_PREFIX = sys._MEIPASS + os.path.sep
        SYS_PREFIXLEN = len(SYS_PREFIX)
        # Only single path is supported, and it must start with sys._MEIPASS.
        pkg_path = os.path.normpath(path[0])
        assert pkg_path.startswith(SYS_PREFIX)
```
So i inserted some debugging code and figured out that:

```
SYS_PREFIX: /var/folders/bd/k__jfmz112sb1_dyz4rxwfzh0000gn/T/
pkg_path: /private/var/folders/bd/k__jfmz112sb1_dyz4rxwfzh0000gn/T/_MEIBqFxGW/cryptoadvance/specter/util/migrations
```
So in combination with the comment above, it seems that MacOS is using "multiple path" and so MacOS is not supported here?
On the other hand, this [link](https://difyel.com/apple/macos/macos-directory-structure/#Other_directories) is claiming that `private/var` is only a symlink to `/var`. So i added this and this seem to have solved it?
```
        if pkg_path.startswith("/private"):
                pkg_path = pkg_path[8:]
        assert pkg_path.startswith(SYS_PREFIX)
```
Is this a valid fix? I might  make a PR if someone can give me some guidance on this.

### Context information (for bug reports)

* Output of `pyinstaller --version`: ```4.8```
* Version of Python: <!-- e.g. 3.9.7 -->
* Platform: OS X
* How you installed Python: brew
* Did you also try this on another platform? Does it work there? --> Yes, works on Linux




Discussion:
> So in combination with the comment above, it seems that MacOS is using "multiple path" and so MacOS is not supported here?

The problem is that `/var` is symlinked to `/private/var` (or vice-versa), which the code does not account for; i.e., it tries to compare the paths as they are, whereas in this case, it should probably be normalizing them.
yes, symlinks. I made a proposal on how to normalize them (in parallel with your comment) and that seem to work if i change it directly in the `pyi_rth_pkgutil.py`. 
So i'm happy to create a PR out of that if it makes sense. However, before i do that, how can i overwrite a rthook? I tried to do it via `--runtime-hook=rthooks/pyi_rth_pkgutil.py` but that didn't work even though i tried it with the same name. Couldn't find anything in the docs about that.
You should just patch your local copy (`path-to-venv-lib/site-packages/PyInstaller/hooks/rthooks/pyi_rth_pkgutil.py`)...

Resolving patch:
diff --git a/PyInstaller/hooks/rthooks/pyi_rth_pkgutil.py b/PyInstaller/hooks/rthooks/pyi_rth_pkgutil.py
--- a/PyInstaller/hooks/rthooks/pyi_rth_pkgutil.py
+++ b/PyInstaller/hooks/rthooks/pyi_rth_pkgutil.py
@@ -52,11 +52,16 @@ def _pyi_pkgutil_iter_modules(path=None, prefix=''):
             yield pkgutil.ModuleInfo(importer, prefix + entry, is_pkg)
     else:
         # Declare SYS_PREFIX locally, to avoid clash with eponymous global symbol from pyi_rth_pkgutil hook.
-        SYS_PREFIX = sys._MEIPASS + os.path.sep
+        #
+        # Use os.path.realpath() to fully resolve any symbolic links in sys._MEIPASS, in order to avoid path mis-matches
+        # when the given search paths also contain symbolic links and are already fully resolved. See #6537 for an
+        # example of such a problem with onefile build on macOS, where the temporary directory is placed under /var,
+        # which is actually a symbolic link to /private/var.
+        SYS_PREFIX = os.path.realpath(sys._MEIPASS) + os.path.sep
         SYS_PREFIXLEN = len(SYS_PREFIX)
 
         for pkg_path in path:
-            pkg_path = os.path.normpath(pkg_path)
+            pkg_path = os.path.realpath(pkg_path)  # Fully resolve the given path, in case it contains symbolic links.
             if not pkg_path.startswith(SYS_PREFIX):
                 # if the path does not start with sys._MEIPASS then it cannot be a bundled package.
                 continue


Test patch:
diff --git a/tests/functional/scripts/pyi_pkgutil_iter_modules.py b/tests/functional/scripts/pyi_pkgutil_iter_modules.py
--- a/tests/functional/scripts/pyi_pkgutil_iter_modules.py
+++ b/tests/functional/scripts/pyi_pkgutil_iter_modules.py
@@ -9,6 +9,7 @@
 # SPDX-License-Identifier: (GPL-2.0-or-later WITH Bootloader-exception)
 #-----------------------------------------------------------------------------
 
+import os
 import sys
 import argparse
 import pkgutil
@@ -27,6 +28,12 @@
     default='',
     help="Optional prefix to pass to iter_modules.",
 )
+parser.add_argument(
+    '--resolve-pkg-path',
+    action='store_true',
+    default=False,
+    help="Resolve symbolic links in package path before passing it to pkgutil.iter_modules.",
+)
 parser.add_argument(
     '--output-file',
     default=None,
@@ -43,7 +50,10 @@
 
 # Iterate over package's module
 package = importlib.import_module(args.package)
-for module in pkgutil.iter_modules(package.__path__, args.prefix):
+pkg_path = package.__path__
+if args.resolve_pkg_path:
+    pkg_path = [os.path.realpath(path) for path in pkg_path]
+for module in pkgutil.iter_modules(pkg_path, args.prefix):
     print("%s;%d" % (module.name, module.ispkg), file=fp)
 
 # Cleanup
diff --git a/tests/functional/test_pkgutil.py b/tests/functional/test_pkgutil.py
--- a/tests/functional/test_pkgutil.py
+++ b/tests/functional/test_pkgutil.py
@@ -49,7 +49,7 @@ def _read_results_file(filename):
     ]
 )
 @pytest.mark.parametrize('archive', ['archive', 'noarchive'])
-def test_pkgutil_iter_modules(package, script_dir, tmpdir, pyi_builder, archive):
+def test_pkgutil_iter_modules(package, script_dir, tmpdir, pyi_builder, archive, resolve_pkg_path=False):
     # Ensure package is available
     if not importable(package.split(".")[0]):
         pytest.skip("Needs " + package)
@@ -80,10 +80,25 @@ def test_pkgutil_iter_modules(package, script_dir, tmpdir, pyi_builder, archive)
             # enable/disable noarchive
             *debug_args,
         ],
-        app_args=[package, '--output-file', out_frozen],
+        app_args=[package, '--output-file', out_frozen] + (['--resolve-pkg-path'] if resolve_pkg_path else [])
     )  # yapf: disable
     # Read results
     results_frozen = _read_results_file(out_frozen)
 
     # Compare
     assert results_unfrozen == results_frozen
+
+
+# Repeat test_pkgutil_iter_modules() test with package path resolving enabled. In this mode, the test script fully
+# resolves the package path before passing it to pkgutil.iter_modules(), reproducing the scenario of #6537 on macOS:
+# the temporary directory used by onefile builds is placed in /var, which is symbolic link to /private/var. Therefore,
+# the resolved package path (/private/var/...) in the frozen application may differ from non-resolved one (/var/...),
+# and our pkgutil.iter_modules() therefore needs to explicitly resolve the given paths and the sys._MEIPASS prefix to
+# ensure proper matching.
+# The test is applicable only to macOS in onefile mode.
+@pytest.mark.darwin
+def test_pkgutil_iter_modules_resolve_pkg_path(script_dir, tmpdir, pyi_builder):
+    if pyi_builder._mode != 'onefile':
+        pytest.skip('The test is applicable only to onefile mode.')
+    # A single combination (altgraph package, archive mode) is enough to check for proper symlink handling.
+    test_pkgutil_iter_modules('altgraph', script_dir, tmpdir, pyi_builder, archive=True, resolve_pkg_path=True)