Repository: pyinstaller/pyinstaller

Issue:
pyInstaller not detecting `regex` as a non-fat binary
<!--
Welcome to the PyInstaller issue tracker! Before creating an issue, please heed the following:

1. This tracker should only be used to report bugs and request features / enhancements to PyInstaller
    - For questions and general support, use the discussions forum.
2. Use the search function before creating a new issue. Duplicates will be closed and directed to
   the original discussion.
3. When making a bug report, make sure you provide all required information. The easier it is for
   maintainers to reproduce, the faster it'll be fixed.
-->

<!-- +++ ONLY TEXT +++ DO NOT POST IMAGES +++ -->

## Description of the issue

I have a project set to build specifically for universal2, on a universal2 build of Python, with a dependency that pulls in [regex](https://pypi.org/project/regex). The build succeeds (with no warnings/errors for a non-fat library), but apparently the [current regex package downloads](https://pypi.org/project/regex/#files) do not include a universal2 build.

This is certainly a regression in the regex build setup (as they claim to support universal2), but it's also a bit troubling that pyInstaller didn't detect the incorrect library platform, as it has done for other libraries that I've had to manually `defuse` to get builds for.

I have verified that even with a completely from-scratch build in a fresh virtual environment, I get ARM-only libraries in my bundled .app:

```
sparkle:Bandcrash.app main$ find . -name '*regex*so' | xargs file
./Contents/Frameworks/regex/_regex.cpython-311-darwin.so: Mach-O 64-bit bundle arm64
```

This is apparently the *only* .so file which isn't fat; all other .so files in the bundle appear as `Mach-O universal binary with 2 architectures`.

### Context information (for bug reports)

* Output of `pyinstaller --version`: 6.1.0
* Version of Python: 3.11.6
* Platform: macOS 14.0
* How you installed Python: python.org/downloads, universal2 build
* Did you also try this on another platform? Does it work there? Build runs fine on other ARM Macs, and Linux/Windows x64 builds function as well (but that's not relevant to this issue). The issue is that the build is succeeding when it shouldn't be.


* try the latest development version, using the following command:

```shell
pip install https://github.com/pyinstaller/pyinstaller/archive/develop.zip
```

* follow *all* the instructions in our "If Things Go Wrong" Guide
  (https://github.com/pyinstaller/pyinstaller/wiki/If-Things-Go-Wrong) and

### Make sure [everything is packaged correctly](https://github.com/pyinstaller/pyinstaller/wiki/How-to-Report-Bugs#make-sure-everything-is-packaged-correctly)

  * [x] start with clean installation
  * [x] use the latest development version
  * [x] Run your frozen program **from a command window (shell)** — instead of double-clicking on it
  * [x] Package your program in **--onedir mode**
  * [x] Package **without UPX**, say: use the option `--noupx` or set `upx=False` in your .spec-file
  * [x] Repackage you application in **verbose/debug mode**. For this, pass the option `--debug` to `pyi-makespec` or `pyinstaller` or use `EXE(..., debug=1, ...)` in your .spec file.


### A minimal example program which shows the error

https://gist.github.com/fluffy-critter/bf2c2e54ca3a636baf9fa82003822755

### Stacktrace / full error message

Error when attempting to run the app bundle on an Intel Mac:

```
ImportError: dlopen(/Applications/Bandcrash.app/Contents/Frameworks/regex/_regex.cpython-311-darwin.so, 0x0002): tried: '/Applications/Bandcrash.app/Contents/Frameworks/regex/_regex.cpython-311-darwin.so' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64h' or 'x86_64')), '/System/Volumes/Preboot/Cryptexes/OS/Applications/Bandcrash.app/Contents/Frameworks/regex/_regex.cpython-311-darwin.so' (no such file), '/Applications/Bandcrash.app/Contents/Frameworks/regex/_regex.cpython-311-darwin.so' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64h' or 'x86_64'))
```

Please also see <https://github.com/pyinstaller/pyinstaller/wiki/How-to-Report-Bugs>
for more about what would use to solve the issue.



Discussion:
Hmmm, I cannot  seem to reproduce this - if I try to build a `universal2` build of a simple `import regex` program under `universal2` python 3.11 (running on x86_64), I get the `PyInstaller.utils.osx.IncompatibleBinaryArchError` for `regex/_regex.cpython-311-darwin.so`, as expected. Same if I try to build bandcrash using its Makefile.

Can you open the `build/Bandcrash/Analysis-00.toc`, and search for `_regex.cpython-311-darwin.so` entry? Was it classified as `EXTENSION`, or something else? We validate the architecture only for `EXTENSION` type...
Interesting, I wonder if there's something different happening on ARM machines, then.

In the .toc I see:

```
  ('regex/_regex.cpython-311-darwin.so',
   '/Users/fluffy/Library/Caches/pypoetry/virtualenvs/bandcrash-nnauGvj1-py3.11/lib/python3.11/site-packages/regex/_regex.cpython-311-darwin.so',
   'EXTENSION'),
```

and if I `file` that I get:

```
/Users/fluffy/Library/Caches/pypoetry/virtualenvs/bandcrash-nnauGvj1-py3.11/lib/python3.11/site-packages/regex/_regex.cpython-311-darwin.so: Mach-O 64-bit bundle arm64
```

Note that I've worked around the issue in Bandcrash by removing the dependency that was pulling in `regex` (since I was using it only for one function that was easily replaced), so the last git commit which would be able to reproduce the issue is `e764e01348c17e06f138c2e79ffa69d5a5e6bd49` (which also has my Makefile rule for detecting the bad architecture in the bundled .app).

I do notice that when building there's a hook for regex:

```
3524 INFO: Loading module hook 'hook-regex.py' from '/Users/fluffy/Library/Caches/pypoetry/virtualenvs/bandcrash-nnauGvj1-py3.11/lib/python3.11/site-packages/_pyinstaller_hooks_contrib/hooks/stdhooks'...
```

I wonder if that might be doing something weird when building on ARM?
Ah, never mind, I just looked at the hook, there's definitely nothing build-platform-dependent in it.
Ah, it's a cache issue. On a clean build, we copy the file to cache directory, try to process it, and raise an error due to invalid architecture. But on the next run, the file is already in cache, so we just take that, which bypasses the check altogether. Can you try adding `--clean` to your PyInstaller command?
Aha, that did it - I got `PyInstaller.utils.osx.IncompatibleBinaryArchError` when running with `--clean`, but didn't on a subsequent run without it.
For what it's worth, adding `--clean` also seems to have fixed the issue that was causing homebrew .so dependencies to leak into the build (which I mentioned on issue #8048).
> For what it's worth, adding `--clean` also seems to have fixed the issue that was causing homebrew .so dependencies to leak into the build (which I mentioned on issue #8048).

Yeah, the caching part has bunch of gotchas, especially if you use PyInstaller with different python environments on the same system. It is safest to use `--clean` in that case (or even in general, unless you are repeating the same build in the same environment).

Resolving patch:
diff --git a/PyInstaller/building/utils.py b/PyInstaller/building/utils.py
--- a/PyInstaller/building/utils.py
+++ b/PyInstaller/building/utils.py
@@ -259,10 +259,6 @@ def process_collected_binary(
         logger.info("Executing: %s", " ".join(cmd))
         subprocess.run(cmd, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
 
-    # Update cache index
-    cache_index[cached_id] = src_digest
-    misc.save_py_data_struct(cache_index_file, cache_index)
-
     # On macOS, we need to modify the given binary's paths to the dependent libraries, in order to ensure they are
     # relocatable and always refer to location within the frozen application. Specifically, we make all dependent
     # library paths relative to @rpath, and set @rpath to point to the top-level application directory, relative to
@@ -300,6 +296,10 @@ def process_collected_binary(
         except Exception as e:
             raise SystemError(f"Failed to process binary {cached_name!r}!") from e
 
+    # Update cache index
+    cache_index[cached_id] = src_digest
+    misc.save_py_data_struct(cache_index_file, cache_index)
+
     return cached_name
 
 


Test patch: