Repository: pyinstaller/pyinstaller

Issue:
Code compiled with 5.6+ doesn't run
<!--
Welcome to the PyInstaller issue tracker! Before creating an issue, please heed the following:

1. This tracker should only be used to report bugs and request features / enhancements to PyInstaller
    - For questions and general support, use the discussions forum.
2. Use the search function before creating a new issue. Duplicates will be closed and directed to
   the original discussion.
3. When making a bug report, make sure you provide all required information. The easier it is for
   maintainers to reproduce, the faster it'll be fixed.
-->

<!-- +++ ONLY TEXT +++ DO NOT POST IMAGES +++ -->

## Description of the issue

Using PyInstaller to compile a Flask app to run in Docker. We're doing a multi-stage build with the compilation happening on the first stage in the official Python 3.10 Debian image, and the binary is run on a second Debian (MiniDeb) image. Manually setting the version of PyInstaller to 5.4 resolves the issue. Did not test with 5.5, issue is present on 5.6 and 5.6.1

### Context information (for bug reports)

* Output of `pyinstaller --version`: ```Not applicable in a non-functional Docker container, but each build pulls the most recent version```
* Version of Python: 3.10
* Platform: Debian (MiniDeb) Bullseye
* How you installed Python: Official Python docker image
* Did you also try this on another platform? Does it work there?: N/A


* try the latest development version, using the following command:

```shell
pip install https://github.com/pyinstaller/pyinstaller/archive/develop.zip
```

* follow *all* the instructions in our "If Things Go Wrong" Guide
  (https://github.com/pyinstaller/pyinstaller/wiki/If-Things-Go-Wrong) and

### Make sure [everything is packaged correctly](https://github.com/pyinstaller/pyinstaller/wiki/How-to-Report-Bugs#make-sure-everything-is-packaged-correctly)

  * [ ] start with clean installation
  * [ ] use the latest development version
  * [ ] Run your frozen program **from a command window (shell)** — instead of double-clicking on it
  * [ ] Package your program in **--onedir mode**
  * [ ] Package **without UPX**, say: use the option `--noupx` or set `upx=False` in your .spec-file
  * [ ] Repackage you application in **verbose/debug mode**. For this, pass the option `--debug` to `pyi-makespec` or `pyinstaller` or use `EXE(..., debug=1, ...)` in your .spec file.


### A minimal example program which shows the error

```

```

### Stacktrace / full error message
```
Traceback (most recent call last):
  File "PyInstaller/hooks/rthooks/pyi_rth_pkgres.py", line 16, in <module>
  File "PyInstaller/loader/pyimod02_importers.py", line 499, in exec_module
  File "pkg_resources/__init__.py", line 32, in <module>
  File "PyInstaller/loader/pyimod02_importers.py", line 499, in exec_module
  File "plistlib.py", line 61, in <module>
  File "PyInstaller/loader/pyimod02_importers.py", line 499, in exec_module
  File "xml/parsers/expat.py", line 4, in <module>
ImportError: libexpat.so.1: cannot open shared object file: No such file or directory
[11] Failed to execute script 'pyi_rth_pkgres' due to unhandled exception!
```

Please also see <https://github.com/pyinstaller/pyinstaller/wiki/How-to-Report-Bugs>
for more about what would use to solve the issue.



Discussion:
Can you give us the names of this images? e.g. is it `python:3.10` or `python:3.10-slim` and `bullseye` or `buster`. I'm guessing that _minideb_ means [this](https://hub.docker.com/r/bitnami/minideb)?
Correct, we're using the Bullseye image of bitnami minideb for the resulting container. And the first stage of the build uses python:3.10-bullseye
Hmm, yeah... 5.4 (and also 5.5) seems to collect expat library as `libexpat.so.1`, while 5.6 collects the fully-versioned `libexpat.so.1.6.12` instead. And `pyexpat` extension is linked against the `.so.1` variant, hence the error.
Caused by af2cd6999c2c4b6a9648fa057290c0cb68efbade according to bisect.
Although I can't see why.
Yeah, I also just bisected it to af2cd6999c2c4b6a9648fa057290c0cb68efbade...
There's probalby a symlink resolution that slipped in somewhere...
Ah, it's because it now constructs destination name based on resolved npth, which resolves the symlink.
Yeah. Previously, `src_path` was set to the resolved variant but not used unless [this conditional](https://github.com/pyinstaller/pyinstaller/commit/af2cd6999c2c4b6a9648fa057290c0cb68efbade#diff-56ffe0db2716631574d6d09d17656b3822352d2f2df656c3a8c04e67e0819513L274) was true. Now `src_path` sets the name either way. That means that both versions are broken really - the old version would have had the same issue if the linked library was in a `sys.path` prefix?
(Also, yet again I find I'm cursing myself for not refactoring away that `npth` variable name :vomiting_face: )
> That means that both versions are broken really - the old version would have had the same issue if the linked library was in a `sys.path` prefix?

Yeah, the design was broken, but it had no effect earlier because we populate preservation paths only on Windows, so the former else branch was always taken on linux and macOS. But after the refactoring, the symlink resolution became effectively active on linux and macOS - which would become a problem once we tried to turn on shared lib directory structure preservation on those OSes.

I think the fix is to simply remove the `.resolve()` call when turning `npth` into `pathlib.Path` [here](https://github.com/pyinstaller/pyinstaller/blob/780c08d7f1827b37bfc94751a411d71451e04a35/PyInstaller/depend/bindepend.py#L293). This way, the filename should stay the same, regardless if the file is collected into sub-directory or into top-level directory.
And, once we start supporting symlinks, we should detect symlinks and collect both the link and its target file.
> I think the fix is to simply remove the .resolve() call when turning npth into pathlib.Path [here](https://github.com/pyinstaller/pyinstaller/blob/780c08d7f1827b37bfc94751a411d71451e04a35/PyInstaller/depend/bindepend.py#L293). This way, the filename should stay the same, regardless if the file is collected into sub-directory or into top-level directory.

Sounds good to me. Looks like another patch release then... Don't think PyInstaller has ever reached an `x.y.2` version before.

Resolving patch:
diff --git a/PyInstaller/depend/bindepend.py b/PyInstaller/depend/bindepend.py
--- a/PyInstaller/depend/bindepend.py
+++ b/PyInstaller/depend/bindepend.py
@@ -237,8 +237,10 @@ def _select_destination_directory(src_filename, parent_dir_preservation_paths):
     # top-level directory... i.e., we must NOT preserve the directory layout in this case.
     if compat.is_win:
         # match <...>/site-packages/pythonwin
+        # The caller might not have resolved the src_filename, so we need to explicitly lower-case the parent_dir.name
+        # before comparing it to account for case variations.
         parent_dir = src_filename.parent
-        if parent_dir.name == "pythonwin" and parent_dir.parent in parent_dir_preservation_paths:
+        if parent_dir.name.lower() == "pythonwin" and parent_dir.parent in parent_dir_preservation_paths:
             # Collect into top-level directory.
             return src_filename.name
 
@@ -290,7 +292,10 @@ def Dependencies(lTOC, xtrapath=None, manifest=None, redirects=None):
             seen.add(npth.upper())
 
             # Try to preserve parent directory structure, if applicable.
-            src_path = pathlib.Path(npth).resolve()
+            # NOTE: do not resolve the source path, because on macOS and linux, it may be a versioned .so (e.g.,
+            # libsomething.so.1, pointing at libsomething.so.1.2.3), and we need to collect it under original
+            # name!
+            src_path = pathlib.Path(npth)
             dst_path = _select_destination_directory(src_path, parent_dir_preservation_paths)
             lTOC.append((str(dst_path), npth, 'BINARY'))
 


Test patch: