Repository: pyinstaller/pyinstaller

Issue:
UPX doesn't seem to get used on executable, linux/pyside6
<!--
Welcome to the PyInstaller issue tracker! Before creating an issue, please heed the following:

1. This tracker should only be used to report bugs and request features / enhancements to PyInstaller
    - For questions and general support, use the mailing list, see
	  <http://www.pyinstaller.org/support.html#mailing-list>
2. Use the search function before creating a new issue. Duplicates will be closed and directed to
   the original discussion.
3. When making a bug report, make sure you provide all required information. The easier it is for
   maintainers to reproduce, the faster it'll be fixed.
-->

<!-- +++ ONLY TEXT +++ DO NOT POST IMAGES +++ -->
<!--
Welcome to the PyInstaller issue tracker! Before creating an issue, please heed the following:

1. This tracker should only be used to report bugs and request features / enhancements to PyInstaller
    - For questions and general support, use the mailing list, see
	  <http://www.pyinstaller.org/support.html#mailing-list>
2. Use the search function before creating a new issue. Duplicates will be closed and directed to
   the original discussion.
3. When making a bug report, make sure you provide all required information. The easier it is for
   maintainers to reproduce, the faster it'll be fixed.
-->

<!-- +++ ONLY TEXT +++ DO NOT POST IMAGES +++ -->

## Description of the issue

UPX says it's available, but it doesn't seem to be used

### Context information (for bug reports)

* Output of `pyinstaller --version`: ```5.0.dev0```
* Version of Python: 3.9.5 <!-- e.g. 3.7 -->
* Platform: Ubuntu 21.04 <!-- e.g GNU/Linux (distribution), Windows (language settings), OS X, FreeBSD -->
* How you installed Python: apt <!-- e.g. python.org/downloads, conda, brew, pyenv, apt, Windows store -->
* Did you also try this on another platform? Does it work there?: Yes, on windows UPX runs fine and the executable is compressed


* try the latest development version, using the following command:

```shell
pip install https://github.com/pyinstaller/pyinstaller/archive/develop.zip
```

* follow *all* the instructions in our "If Things Go Wrong" Guide
  (https://github.com/pyinstaller/pyinstaller/wiki/If-Things-Go-Wrong) and

### Make sure [everything is packaged correctly](https://github.com/pyinstaller/pyinstaller/wiki/How-to-Report-Bugs#make-sure-everything-is-packaged-correctly)

  * [x] start with clean installation
  * [x] use the latest development version
  * [x] Run your frozen program **from a command window (shell)** — instead of double-clicking on it
  * [ ] Package your program in **--onedir mode**
  * [x] Package **without UPX**, say: use the option `--noupx` or set `upx=False` in your .spec-file
  * [x] Repackage you application in **verbose/debug mode**. For this, pass the option `--debug` to `pyi-makespec` or `pyinstaller` or use `EXE(..., debug=1, ...)` in your .spec file.


File Sizes:
No `--noupx` flag: 203627816
With `--noupx` flag: 203627736

Logs:
```
45 INFO: PyInstaller: 5.0.dev0
46 INFO: Python: 3.9.5
48 INFO: Platform: Linux-5.11.0-22-generic-x86_64-with-glibc2.33
49 INFO: wrote /home/anitracker/anitracker.spec
61 INFO: UPX is available.
62 INFO: Removing temporary files and cleaning cache in /home/anitracker/.cache/pyinstaller
73 INFO: Extending PYTHONPATH with paths
['/home/anitracker', '/home/anitracker']
82 INFO: checking Analysis
82 INFO: Building Analysis because Analysis-00.toc is non existent
82 INFO: Initializing module dependency graph...
83 INFO: Caching module graph hooks...
87 INFO: Analyzing base_library.zip ...
1153 INFO: Processing pre-find module path hook distutils from '/home/anitracker/.env/lib/python3.9/site-packages/PyInstaller/hooks/pre_find_module_path/hook-distutils.py'.
1154 INFO: distutils: retargeting to non-venv dir '/usr/lib/python3.9'
2618 INFO: Caching module dependency graph...
2687 INFO: running Analysis Analysis-00.toc
2707 INFO: Analyzing /home/anitracker/anitracker/__main__.py
2777 INFO: Processing pre-safe import module hook urllib3.packages.six.moves from '/home/anitracker/.env/lib/python3.9/site-packages/PyInstaller/hooks/pre_safe_import_module/hook-urllib3.packages.six.moves.py'.
4195 INFO: Processing module hooks...
4195 INFO: Loading module hook 'hook-certifi.py' from '/home/anitracker/.env/lib/python3.9/site-packages/_pyinstaller_hooks_contrib/hooks/stdhooks'...
4202 INFO: Loading module hook 'hook-cryptography.py' from '/home/anitracker/.env/lib/python3.9/site-packages/_pyinstaller_hooks_contrib/hooks/stdhooks'...
4338 INFO: Loading module hook 'hook-sysconfig.py' from '/home/anitracker/.env/lib/python3.9/site-packages/PyInstaller/hooks'...
4344 INFO: Loading module hook 'hook-PySide6.QtCore.py' from '/home/anitracker/.env/lib/python3.9/site-packages/PyInstaller/hooks'...
4573 INFO: Loading module hook 'hook-xml.py' from '/home/anitracker/.env/lib/python3.9/site-packages/PyInstaller/hooks'...
4599 INFO: Loading module hook 'hook-multiprocessing.util.py' from '/home/anitracker/.env/lib/python3.9/site-packages/PyInstaller/hooks'...
4601 INFO: Loading module hook 'hook-difflib.py' from '/home/anitracker/.env/lib/python3.9/site-packages/PyInstaller/hooks'...
4603 INFO: Loading module hook 'hook-lib2to3.py' from '/home/anitracker/.env/lib/python3.9/site-packages/PyInstaller/hooks'...
4609 INFO: Loading module hook 'hook-distutils.py' from '/home/anitracker/.env/lib/python3.9/site-packages/PyInstaller/hooks'...
4610 INFO: Loading module hook 'hook-encodings.py' from '/home/anitracker/.env/lib/python3.9/site-packages/PyInstaller/hooks'...
4638 INFO: Loading module hook 'hook-PySide6.py' from '/home/anitracker/.env/lib/python3.9/site-packages/PyInstaller/hooks'...
4645 INFO: Loading module hook 'hook-pickle.py' from '/home/anitracker/.env/lib/python3.9/site-packages/PyInstaller/hooks'...
4648 INFO: Loading module hook 'hook-PySide6.QtGui.py' from '/home/anitracker/.env/lib/python3.9/site-packages/PyInstaller/hooks'...
4703 INFO: Loading module hook 'hook-PySide6.QtWidgets.py' from '/home/anitracker/.env/lib/python3.9/site-packages/PyInstaller/hooks'...
4754 INFO: Loading module hook 'hook-PySide6.QtNetwork.py' from '/home/anitracker/.env/lib/python3.9/site-packages/PyInstaller/hooks'...
4784 INFO: Loading module hook 'hook-importlib_metadata.py' from '/home/anitracker/.env/lib/python3.9/site-packages/PyInstaller/hooks'...
4785 INFO: Loading module hook 'hook-heapq.py' from '/home/anitracker/.env/lib/python3.9/site-packages/PyInstaller/hooks'...
4788 INFO: Loading module hook 'hook-distutils.util.py' from '/home/anitracker/.env/lib/python3.9/site-packages/PyInstaller/hooks'...
4789 INFO: Loading module hook 'hook-xml.etree.cElementTree.py' from '/home/anitracker/.env/lib/python3.9/site-packages/PyInstaller/hooks'...
4790 INFO: Loading module hook 'hook-PySide6.QtQml.py' from '/home/anitracker/.env/lib/python3.9/site-packages/PyInstaller/hooks'...
4855 INFO: Looking for ctypes DLLs
4888 INFO: Analyzing run-time hooks ...
4892 INFO: Including run-time hook '/home/anitracker/.env/lib/python3.9/site-packages/PyInstaller/hooks/rthooks/pyi_rth_multiprocessing.py'
4895 INFO: Including run-time hook '/home/anitracker/.env/lib/python3.9/site-packages/PyInstaller/hooks/rthooks/pyi_rth_pyside6.py'
4896 INFO: Including run-time hook '/home/anitracker/.env/lib/python3.9/site-packages/_pyinstaller_hooks_contrib/hooks/rthooks/pyi_rth_certifi.py'
4901 INFO: Looking for dynamic libraries
6398 INFO: Looking for eggs
6398 INFO: Python library not in binary dependencies. Doing additional searching...
6436 INFO: Using Python library /lib/x86_64-linux-gnu/libpython3.9.so.1.0
6441 INFO: Warnings written to /home/anitracker/build/anitracker/warn-anitracker.txt
6464 INFO: Graph cross-reference written to /home/anitracker/build/anitracker/xref-anitracker.html
6487 INFO: checking PYZ
6487 INFO: Building PYZ because PYZ-00.toc is non existent
6487 INFO: Building PYZ (ZlibArchive) /home/anitracker/build/anitracker/PYZ-00.pyz
6814 INFO: Building PYZ (ZlibArchive) /home/anitracker/build/anitracker/PYZ-00.pyz completed successfully.
6821 INFO: checking PKG
6821 INFO: Building PKG because PKG-00.toc is non existent
6821 INFO: Building PKG (CArchive) PKG-00.pkg
54607 INFO: Building PKG (CArchive) PKG-00.pkg completed successfully.
54623 INFO: Bootloader /home/anitracker/.env/lib/python3.9/site-packages/PyInstaller/bootloader/Linux-64bit/run
54624 INFO: checking EXE
54624 INFO: Building EXE because EXE-00.toc is non existent
54624 INFO: Building EXE from EXE-00.toc
54627 INFO: Appending archive to ELF section in EXE /home/anitracker/dist/anitracker
55324 INFO: Building EXE from EXE-00.toc completed successfully.
```


Discussion:
It seems `upx` is actually used by `PyInstaller` only on Windows:

https://github.com/pyinstaller/pyinstaller/blob/0f2b2e921433ab5a510c7efdb21d9c1d7cfbc645/PyInstaller/building/utils.py#L181-L182

You could try editing your `site-packages/PyInstaller/building/utils.py` and remove that `and (is_win or is_cygwin)`, which should enable `upx` on any OS.

Although it seems that using `upx` on linux breaks even a simple CLI hello-world frozen application...
Alright, based on that information I'd assume there was some type of regression with UPX that caused this failure. I'll test around some versions and see if I can figure what/if any work on linux, and comment with whatever I find
I had a walk through the git blame trying to find a reason.

It was last touched here:
* https://github.com/pyinstaller/pyinstaller/blob/257800e94034bd8bb4ce00bc0f3898cfa2f1064f/PyInstaller/building/api.py#L258

It's been moved around and refactored quite a bit before then.
* https://github.com/pyinstaller/pyinstaller/commit/4cf8d8217e1b5996f6b73b10d0f3717bddd296c2
* https://github.com/pyinstaller/pyinstaller/commit/97529018433e855f1d5a3d2c153cf1b78e645bf3#diff-02f4b0ba41d4545149eee743b8e5ddd8bf85691a0ef7602d96afd88f758afbd1R1267

At this point that code snippet duplicates. There seemed to be two copies of that check so the blame is now in pairs.

* https://github.com/pyinstaller/pyinstaller/blob/b55d07312376ed761f75b2111534cec90468a71c/PyInstaller/build.py#L964
  https://github.com/pyinstaller/pyinstaller/blob/b55d07312376ed761f75b2111534cec90468a71c/PyInstaller/build.py#L1267

* https://github.com/pyinstaller/pyinstaller/commit/038fb9cdc1b59cf52b2a1090698241b8a69fb4d0#diff-02f4b0ba41d4545149eee743b8e5ddd8bf85691a0ef7602d96afd88f758afbd1L798
  https://github.com/pyinstaller/pyinstaller/commit/038fb9cdc1b59cf52b2a1090698241b8a69fb4d0#diff-02f4b0ba41d4545149eee743b8e5ddd8bf85691a0ef7602d96afd88f758afbd1L1141

Then it unifies again.

* https://github.com/pyinstaller/pyinstaller/blame/deb4a0119409519009684f15c5a49093b44f6190/Build.py#L621

And finally, we hit the dawn of PyInstaller on git:
* https://github.com/pyinstaller/pyinstaller/commit/d2e8902ae197d4b9f8295c0b1e91f2330a15b1e2#diff-a3d99d203acd26ccf6d3ff8123f0885fdf32117ace126d465f6b45832c341dd5R367

So unless Rasky remembers a decision he must have made at least 16 years ago back when PyInstaller was called Python Installer and ran on Python 1.x, I guess we'll never know...

The odd thing is that even though this has been effectively disabled on non-Windows since forever, there are some parts that explicitly try to handle upx on linux (e.g., the [part that omits `--lzma` argument](https://github.com/pyinstaller/pyinstaller/blob/ab81fe39b11bc12216f86e85fb899ff13379c069/PyInstaller/building/utils.py#L283-L286)).

----

I tried writing a simple C program that `dlopen`s the given shared library, and if I try to feed it libpython that's been processed by `upx` (3.96), it also segfaults (at least on my Fedora 34 machine) whereas the normal one is dlopened as expected. So unless we can figure out the cause of that, there's a very slim chance of upx being enabled on linux...

Resolving patch:
diff --git a/PyInstaller/building/utils.py b/PyInstaller/building/utils.py
--- a/PyInstaller/building/utils.py
+++ b/PyInstaller/building/utils.py
@@ -16,7 +16,7 @@
 import glob
 import hashlib
 import os
-import os.path
+import pathlib
 import pkgutil
 import platform
 import shutil
@@ -167,8 +167,25 @@ def checkCache(
         strip = True
     else:
         strip = False
+
+    # Disable UPX on non-Windows. Using UPX (3.96) on modern Linux shared libraries (for example, the python3.x.so
+    # shared library) seems to result in segmentation fault when they are dlopen'd. This happens in recent versions
+    # of Fedora and Ubuntu linux, as well as in Alpine containers. On Mac OS, UPX (3.96) fails with
+    # UnknownExecutableFormatException on most .dylibs (and interferes with code signature on other occassions). And
+    # even when it would succeed, compressed libraries cannot be (re)signed due to failed strict validation.
+    upx = upx and (is_win or is_cygwin)
+
+    # Match against provided UPX exclude patterns.
     upx_exclude = upx_exclude or []
-    upx = (upx and (is_win or is_cygwin) and os.path.normcase(os.path.basename(fnm)) not in upx_exclude)
+    if upx:
+        fnm_path = pathlib.PurePath(fnm)
+        for upx_exclude_entry in upx_exclude:
+            # pathlib.PurePath.match() matches from right to left, and supports * wildcard, but does not support the
+            # "**" syntax for directory recursion. Case sensitivity follows the OS default.
+            if fnm_path.match(upx_exclude_entry):
+                logger.info("Disabling UPX for %s due to match in exclude pattern: %s", fnm, upx_exclude_entry)
+                upx = False
+                break
 
     # Load cache index.
     # Make cachedir per Python major/minor version.


Test patch: