Repository: pyinstaller/pyinstaller

Issue:
Qt6 (PyQt/PySide) submodules fail DDL load on import if '__init__.py' is present on Windows
<!--
Welcome to the PyInstaller issue tracker! Before creating an issue, please heed the following:

1. This tracker should only be used to report bugs and request features / enhancements to PyInstaller
    - For questions and general support, use the discussions forum.
2. Use the search function before creating a new issue. Duplicates will be closed and directed to
   the original discussion.
3. When making a bug report, make sure you provide all required information. The easier it is for
   maintainers to reproduce, the faster it'll be fixed.
-->

<!-- +++ ONLY TEXT +++ DO NOT POST IMAGES +++ -->

## Description of the issue

Freezing a script that imports a Qt6 submodule via binding (either PyQt6 or PySide6) fails to import the submodule when running the resulting executable if an \_\_init\_\_.py file is present in the source directory, and PyInstaller was ran from that source directory. There are no build errors or warnings.

This doesn't seem to happen with pyinstaller 5.3 though, after downgrading it with ```pip install pyinstaller==5.3``` the frozen program executes without error.

Walkthrough to recreate the error:

1. ```python -m venv test_env```
2. ```test_env\Scripts\activate.bat```
3. ```pip install https://github.com/pyinstaller/pyinstaller/archive/develop.zip```
4. ```pip install PyQt6```
5. ```fsutil file createnew __init__.py 0```
6. ```echo import PyQt6.QtWidgets > test.py```
7. ```python -m PyInstaller test.py```
8. ```dist\test\test.exe```

Results in:
```
Traceback (most recent call last):
  File "test.py", line 1, in <module>
    import PyQt6.QtWidgets
ImportError: DLL load failed while importing QtWidgets: The specified module could not be found.
[5444] Failed to execute script 'test' due to unhandled exception!
```

Though the relevant DLLs do seem to be in the ```dist\test\PyQt6\Qt6\bin``` folder.

### Context information (for bug reports)

* Output of `pyinstaller --version`: ```5.5```
* Version of Python: 3.10.8
* Platform: Windows 10 (en) <!-- e.g GNU/Linux (distribution), Windows (language settings), OS X, FreeBSD -->
* How you installed Python: python.org provided installer <!-- e.g. python.org/downloads, conda, brew, pyenv, apt, Windows store -->
* Did you also try this on another platform? Does it work there?

Tried it on MacOS and Linux, but it only happened for me on Windows.

* try the latest development version, using the following command:

```shell
pip install https://github.com/pyinstaller/pyinstaller/archive/develop.zip
```

Same result as 5.5.

* follow *all* the instructions in our "If Things Go Wrong" Guide
  (https://github.com/pyinstaller/pyinstaller/wiki/If-Things-Go-Wrong) and

### Make sure [everything is packaged correctly](https://github.com/pyinstaller/pyinstaller/wiki/How-to-Report-Bugs#make-sure-everything-is-packaged-correctly)

  * [x] start with clean installation
  * [x] use the latest development version
  * [x] Run your frozen program **from a command window (shell)** — instead of double-clicking on it
  * [x] Package your program in **--onedir mode**
  * [x] Package **without UPX**, say: use the option `--noupx` or set `upx=False` in your .spec-file
  * [x] Repackage you application in **verbose/debug mode**. For this, pass the option `--debug` to `pyi-makespec` or `pyinstaller` or use `EXE(..., debug=1, ...)` in your .spec file.


### A minimal example program which shows the error

```
import PyQt6.QtCore
# Or:
#import PyQt6.QtWidgets
#import PySide6.QtCore
```

### Stacktrace / full error message


```
[11684] LOADER: Running importtest.py
# PyQt6 not found in PYZ
# code object from 'D:\\isolation\\dist\\importtest\\PyQt6\\__init__.pyc'
import 'PyQt6' # <_frozen_importlib_external.SourcelessFileLoader object at 0x000002AB9E744190>
# PyQt6.QtCore not found in PYZ
Traceback (most recent call last):
  File "importtest.py", line 1, in <module>
    import PyQt6.QtCore
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 674, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 571, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 1176, in create_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
ImportError: DLL load failed while importing QtCore: The specified module could not be found.
[11684] Failed to execute script 'importtest' due to unhandled exception!
```



Discussion:
Can you archive the `dist` and `build` directory of the failed build, and upload them somewhere so I can take a look at them? Thanks!
Sure thing: https://drive.proton.me/urls/D400SK1D4M#hX2OgBKia2cy
Ah yes, the python DLLs are mistakenly collected into `Pythons\Python-3.10.8` subdirectory instead of the root directory. This is a combination of #7028 (PyInstaller 5.4), the fact that you have an `__init__.py` next to the entry-point script, and the layout of your project directory w.r.t. the location of your python installation.

If we find an `__init__.py` next to entry-point script, we add the parent directory to `sys.path` (under the assumption that it is a package and that some imports from it need to be resolved). In your case, the test code is located in `D:\isotest2`, so we add `D:\` to `sys.path`. And then, when we check if we need to preserve the directory layout of for example `D:\\Pythons\\Python-3.10.8\\python3.dll'`, we mistakenly believe that we do, because the path is rooted in one of the paths of `sys.path` (the `D:\` one). 

The error should go away if you move your test directory one level deeper; e.g., `d:\myprojects\isotest2` (and if `myprojects` does not contain the `__init__.py`) - is that the case?

But I think we should probably handle such corner cases anyway, either by not augmenting `sys.path` with drive root, or with additional safeguards in the new layout-preservation code...
That works, after moving it into a sub-directory it runs normally 🐈🎉

Resolving patch:
diff --git a/PyInstaller/depend/bindepend.py b/PyInstaller/depend/bindepend.py
--- a/PyInstaller/depend/bindepend.py
+++ b/PyInstaller/depend/bindepend.py
@@ -189,20 +189,23 @@ def _get_paths_for_parent_directory_preservation():
     frozen application's top-level directory.
     """
 
-    # NOTE: using all paths from `sys.path` entries instead of just `site.getsitepackages() implicitly ensures that we
-    # still collect DLLs from python's "DLLs" sub-directory into top-level directory (because "DLLs" directory is among
-    # the listed paths). On the other hand, `site.getsitepackages()` contains only python's top-level directory and its
-    # "lib/site-packages" subdirectory; so binaries from "DLLs" sub-directory would end up in sub-directory, due to
-    # being relative to python's top directory (which we would need to remove from the list).
-    orig_paths = sys.path
-
-    # Explicitly ignore top-level sys.prefix and sys.base_prefix, to prevent accidental matching of unforeseen
-    # auxiliary sub-directories. For example, under Windows anaconda, shared libraries are located in
-    # `sys.base_prefix`/Library/bin (which is not in `sys.path`), but we need to collect those to the top-level
-    # application directory.
+    # Use only site-packages paths. We have no control over contents of `sys.path`, so using all paths from that may
+    # lead to unintended behavior in corner cases. For example, if `sys.path` contained the drive root (see #7028),
+    # all paths that do not match some other sub-path rooted in that drive will end up recognized as relative to the
+    # drive root. In such case, any DLL collected from `c:\Windows\system32` will be collected into `Windows\system32`
+    # sub-directory; ucrt DLLs collected from MSVC or Windows SDK installed in `c:\Program Files\...` will end up
+    # collected into `Program Files\...` subdirectory; etc.
     #
-    # With top-level python directory excluded, we could switch from `sys.path` to `site.getsitepackages()`, but it
-    # is probably better to use `sys.path` lest it contains any extra user-defined paths.
+    # On the other hand, the DLL parent directory preservation is primarily aimed at packages installed via PyPI
+    # wheels, which are typically installed into site-packages. Therefore, limiting the directory preservation for
+    # shared libraries collected from site-packages should do the trick, and should be reasonably safe.
+    import site
+
+    orig_paths = site.getsitepackages()
+    orig_paths.append(site.getusersitepackages())
+
+    # Explicitly excluded paths. `site.getsitepackages` seems to include `sys.prefix`, which we need to exclude, to
+    # avoid issue swith DLLs in its sub-directories.
     excluded_paths = {
         pathlib.Path(sys.base_prefix).resolve(),
         pathlib.Path(sys.prefix).resolve(),
@@ -210,8 +213,10 @@ def _get_paths_for_parent_directory_preservation():
 
     paths = []
     for path in orig_paths:
+        if not path:
+            continue
         path = pathlib.Path(path).resolve()
-        # Filter out non-directories (e.g., /path/to/python3x.zip)
+        # Filter out non-directories (e.g., /path/to/python3x.zip) or non-existent paths
         if not path.is_dir():
             continue
         # Filter out explicitly excluded paths
@@ -226,6 +231,27 @@ def _get_paths_for_parent_directory_preservation():
     return paths
 
 
+def _select_destination_directory(src_filename, parent_dir_preservation_paths):
+    # Special handling for pywin32 on Windows, because its .pyd extensions end up linking each other, but, due to
+    # sys.path modifications the packages perform, they all end up as top-modules and should be collected into
+    # top-level directory... i.e., we must NOT preserve the directory layout in this case.
+    if compat.is_win:
+        # match <...>/site-packages/pythonwin
+        parent_dir = src_filename.parent
+        if parent_dir.name == "pythonwin" and parent_dir.parent in parent_dir_preservation_paths:
+            # Collect into top-level directory.
+            return src_filename.name
+
+    # Check parent directory preservation paths
+    for parent_dir_preservation_path in parent_dir_preservation_paths:
+        if parent_dir_preservation_path in src_filename.parents:
+            # Collect into corresponding sub-directory.
+            return src_filename.relative_to(parent_dir_preservation_path)
+
+    # Collect into top-level directory.
+    return src_filename.name
+
+
 def Dependencies(lTOC, xtrapath=None, manifest=None, redirects=None):
     """
     Expand LTOC to include all the closure of binary dependencies.
@@ -265,17 +291,8 @@ def Dependencies(lTOC, xtrapath=None, manifest=None, redirects=None):
 
             # Try to preserve parent directory structure, if applicable.
             src_path = pathlib.Path(npth).resolve()
-            for parent_dir_preservation_path in parent_dir_preservation_paths:
-                if parent_dir_preservation_path in src_path.parents:
-                    # Collect into corresponding sub-directory.
-                    rel_lib_path = src_path.relative_to(parent_dir_preservation_path)
-                    lTOC.append((str(rel_lib_path), npth, 'BINARY'))
-                    # TODO: once this codepath is enabled on linux and macOS, add a symlink to the binary into the
-                    # application top-level directory.
-                    break
-            else:
-                # Collect into top-level directory.
-                lTOC.append((lib, npth, 'BINARY'))
+            dst_path = _select_destination_directory(src_path, parent_dir_preservation_paths)
+            lTOC.append((str(dst_path), npth, 'BINARY'))
 
     return lTOC
 


Test patch: