Repository: pyinstaller/pyinstaller

Issue:
Bad CPU Type on Apple Silicon
## Description of the issue
PyInstaller reports `Bad CPU type in executable` when run on Apple M1 Silicon (Big Sur) at various points during building and fails to build executable.

## Context information (for bug reports)
* Output of `pyinstaller --version`: ```4.2```
* Version of Python: e.g. Python 3.9.2
* Platform: MacOS Big Sur V 11.1 (20C69) M1 2020 Silicon 
* Same command works with on Catalina, Intel Silicon on several other machines with python 3.9.2 and pyinstaller 4.2
* Results from development version `5.0.dev0`: Same results
* "If Things Go Wrong" Guide: does not appear to apply to this situation

### Make sure [everything is packaged correctly](https://github.com/pyinstaller/pyinstaller/wiki/How-to-Report-Bugs#make-sure-everything-is-packaged-correctly)

  * [x] start with clean installation
  * [x] use the latest development version
  * [x] Run your frozen program **from a command window (shell)** — instead of double-clicking on it
  * [x] Package your program in **--onedir mode**
  * [x] Package **without UPX**, say: use the option `--noupx` or set `upx=False` in your .spec-file
  * [ ] Repackage you application in **verbose/debug mode**. For this, pass the option `--debug` to `pyi-makespec` or `pyinstaller` or use `EXE(..., debug=1, ...)` in your .spec file. 
      * I am unsure how to do this
     
### A minimal example program which shows the error

```
print('hello world!')
```

### Stacktrace / full error message of minimal example:
A minimal example does not fail to build, but does generate errors 
```
$ pipenv run pyinstaller --onefile --noconfirm --clean --add-data Help.md:. --add-data logging_cfg.ini:. --add-data createFolders.ini:. --exclude-module iPython --noupx hello.py
18 INFO: PyInstaller: 5.0.dev0
19 INFO: Python: 3.9.2
24 INFO: Platform: macOS-11.1-arm64-arm-64bit
24 INFO: wrote /Users/aciuffo/Documents/src/portfolioCreator/create_folders/hello.spec
26 INFO: UPX is not available.
26 INFO: Removing temporary files and cleaning cache in /Users/aciuffo/Library/Application Support/pyinstaller
29 INFO: Extending PYTHONPATH with paths
['/Users/aciuffo/Documents/src/portfolioCreator',
 '/Users/aciuffo/Documents/src/portfolioCreator/create_folders']
31 INFO: checking Analysis
31 INFO: Building Analysis because Analysis-00.toc is non existent
31 INFO: Initializing module dependency graph...
32 INFO: Caching module graph hooks...
34 WARNING: Several hooks defined for module 'win32ctypes.core'. Please take care they do not conflict.
35 INFO: Analyzing base_library.zip ...
arch: posix_spawnp: /Users/aciuffo/.local/share/virtualenvs/portfolioCreator-ahiuVo4K/bin/python: Bad CPU type in executable
arch: posix_spawnp: /Users/aciuffo/.local/share/virtualenvs/portfolioCreator-ahiuVo4K/bin/python: Bad CPU type in executable
1056 INFO: Processing pre-find module path hook distutils from '/Users/aciuffo/.local/share/virtualenvs/portfolioCreator-ahiuVo4K/lib/python3.9/site-packages/PyInstaller/hooks/pre_find_module_path/hook-distutils.py'.
1056 INFO: distutils: retargeting to non-venv dir '/opt/homebrew/Cellar/python@3.9/3.9.2_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9'
2230 INFO: Caching module dependency graph...
2272 INFO: running Analysis Analysis-00.toc
2275 INFO: Analyzing /Users/aciuffo/Documents/src/portfolioCreator/create_folders/hello.py
2276 INFO: Processing module hooks...
2276 INFO: Loading module hook 'hook-xml.etree.cElementTree.py' from '/Users/aciuffo/.local/share/virtualenvs/portfolioCreator-ahiuVo4K/lib/python3.9/site-packages/PyInstaller/hooks'...
2276 INFO: Loading module hook 'hook-lib2to3.py' from '/Users/aciuffo/.local/share/virtualenvs/portfolioCreator-ahiuVo4K/lib/python3.9/site-packages/PyInstaller/hooks'...
2282 INFO: Loading module hook 'hook-_tkinter.py' from '/Users/aciuffo/.local/share/virtualenvs/portfolioCreator-ahiuVo4K/lib/python3.9/site-packages/PyInstaller/hooks'...
arch: posix_spawnp: /Users/aciuffo/.local/share/virtualenvs/portfolioCreator-ahiuVo4K/bin/python: Bad CPU type in executable
arch: posix_spawnp: /Users/aciuffo/.local/share/virtualenvs/portfolioCreator-ahiuVo4K/bin/python: Bad CPU type in executable
2289 ERROR: Tcl/Tk improperly installed on this system.
2289 INFO: Loading module hook 'hook-encodings.py' from '/Users/aciuffo/.local/share/virtualenvs/portfolioCreator-ahiuVo4K/lib/python3.9/site-packages/PyInstaller/hooks'...
arch: posix_spawnp: /Users/aciuffo/.local/share/virtualenvs/portfolioCreator-ahiuVo4K/bin/python: Bad CPU type in executable
2292 INFO: Loading module hook 'hook-distutils.util.py' from '/Users/aciuffo/.local/share/virtualenvs/portfolioCreator-ahiuVo4K/lib/python3.9/site-packages/PyInstaller/hooks'...
2293 INFO: Excluding import of lib2to3.refactor from module distutils.util
2293 INFO: Loading module hook 'hook-pickle.py' from '/Users/aciuffo/.local/share/virtualenvs/portfolioCreator-ahiuVo4K/lib/python3.9/site-packages/PyInstaller/hooks'...
2293 INFO: Excluding import of argparse from module pickle
2293 INFO: Loading module hook 'hook-heapq.py' from '/Users/aciuffo/.local/share/virtualenvs/portfolioCreator-ahiuVo4K/lib/python3.9/site-packages/PyInstaller/hooks'...
2294 INFO: Excluding import of doctest from module heapq
2294 INFO: Loading module hook 'hook-difflib.py' from '/Users/aciuffo/.local/share/virtualenvs/portfolioCreator-ahiuVo4K/lib/python3.9/site-packages/PyInstaller/hooks'...
2294 INFO: Excluding import of doctest from module difflib
2294 INFO: Loading module hook 'hook-multiprocessing.util.py' from '/Users/aciuffo/.local/share/virtualenvs/portfolioCreator-ahiuVo4K/lib/python3.9/site-packages/PyInstaller/hooks'...
2294 INFO: Excluding import of test from module multiprocessing.util
2294 INFO: Excluding import of test.support from module multiprocessing.util
2294 INFO: Loading module hook 'hook-sysconfig.py' from '/Users/aciuffo/.local/share/virtualenvs/portfolioCreator-ahiuVo4K/lib/python3.9/site-packages/PyInstaller/hooks'...
2299 INFO: Loading module hook 'hook-xml.py' from '/Users/aciuffo/.local/share/virtualenvs/portfolioCreator-ahiuVo4K/lib/python3.9/site-packages/PyInstaller/hooks'...
2321 INFO: Loading module hook 'hook-distutils.py' from '/Users/aciuffo/.local/share/virtualenvs/portfolioCreator-ahiuVo4K/lib/python3.9/site-packages/PyInstaller/hooks'...
2327 INFO: Looking for ctypes DLLs
2340 INFO: Analyzing run-time hooks ...
2342 INFO: Including run-time hook '/Users/aciuffo/.local/share/virtualenvs/portfolioCreator-ahiuVo4K/lib/python3.9/site-packages/PyInstaller/hooks/rthooks/pyi_rth_multiprocessing.py'
2344 INFO: Looking for dynamic libraries
2368 INFO: Looking for eggs
2368 INFO: Using Python library /opt/homebrew/Cellar/python@3.9/3.9.2_1/Frameworks/Python.framework/Versions/3.9/Python
2369 INFO: Warnings written to /Users/aciuffo/Documents/src/portfolioCreator/create_folders/build/hello/warn-hello.txt
2378 INFO: Graph cross-reference written to /Users/aciuffo/Documents/src/portfolioCreator/create_folders/build/hello/xref-hello.html
2382 INFO: Appending 'datas' from .spec
2383 INFO: checking PYZ
2383 INFO: Building PYZ because PYZ-00.toc is non existent
2383 INFO: Building PYZ (ZlibArchive) /Users/aciuffo/Documents/src/portfolioCreator/create_folders/build/hello/PYZ-00.pyz
2586 INFO: Building PYZ (ZlibArchive) /Users/aciuffo/Documents/src/portfolioCreator/create_folders/build/hello/PYZ-00.pyz completed successfully.
2589 INFO: checking PKG
2589 INFO: Building PKG because PKG-00.toc is non existent
2589 INFO: Building PKG (CArchive) PKG-00.pkg
3847 INFO: Building PKG (CArchive) PKG-00.pkg completed successfully.
3848 INFO: Bootloader /Users/aciuffo/.local/share/virtualenvs/portfolioCreator-ahiuVo4K/lib/python3.9/site-packages/PyInstaller/bootloader/Darwin-64bit-arm/run
3848 INFO: checking EXE
3848 INFO: Building EXE because EXE-00.toc is non existent
3848 INFO: Building EXE from EXE-00.toc
3848 INFO: Appending archive to EXE /Users/aciuffo/Documents/src/portfolioCreator/create_folders/dist/hello
3852 INFO: Fixing EXE for code signing /Users/aciuffo/Documents/src/portfolioCreator/create_folders/dist/hello
3854 INFO: Building EXE from EXE-00.toc completed successfully.
```

### Output of executable:
```
$ ./dist/hello                                                      (master) ✗
[39586] Error loading Python lib '/var/folders/sg/vqd02czj31sbb6g659kyvdjr00011b/T/_MEIAVkJlt/Python': dlopen: dlopen(/var/folders/sg/vqd02czj31sbb6g659kyvdjr00011b/T/_MEIAVkJlt/Python, 10): no suitable image found.  Did find:
	/var/folders/sg/vqd02czj31sbb6g659kyvdjr00011b/T/_MEIAVkJlt/Python: mach-o, but wrong architecture
	/private/var/folders/sg/vqd02czj31sbb6g659kyvdjr00011b/T/_MEIAVkJlt/Python: mach-o, but wrong architecture
```


## Output when building full program:
```
$ pipenv run pyinstaller --onedir --noconfirm --clean --add-data Help.md:. --add-data logging_cfg.ini:. --add-data createFolders.ini:. --exclude-module iPython --noupx createFolders.py
80 INFO: PyInstaller: 5.0.dev0
80 INFO: Python: 3.9.2
92 INFO: Platform: macOS-11.1-arm64-arm-64bit
94 INFO: wrote /Users/aciuffo/Documents/src/portfolioCreator/create_folders/createFolders.spec
96 INFO: UPX is not available.
96 INFO: Removing temporary files and cleaning cache in /Users/aciuffo/Library/Application Support/pyinstaller
118 INFO: Extending PYTHONPATH with paths
['/Users/aciuffo/Documents/src/portfolioCreator',
 '/Users/aciuffo/Documents/src/portfolioCreator/create_folders']
149 INFO: checking Analysis
149 INFO: Building Analysis because Analysis-00.toc is non existent
149 INFO: Initializing module dependency graph...
150 INFO: Caching module graph hooks...
152 WARNING: Several hooks defined for module 'win32ctypes.core'. Please take care they do not conflict.
153 INFO: Analyzing base_library.zip ...
arch: posix_spawnp: /Users/aciuffo/.local/share/virtualenvs/portfolioCreator-ahiuVo4K/bin/python: Bad CPU type in executable
arch: posix_spawnp: /Users/aciuffo/.local/share/virtualenvs/portfolioCreator-ahiuVo4K/bin/python: Bad CPU type in executable
1130 INFO: Processing pre-find module path hook distutils from '/Users/aciuffo/.local/share/virtualenvs/portfolioCreator-ahiuVo4K/lib/python3.9/site-packages/PyInstaller/hooks/pre_find_module_path/hook-distutils.py'.
1131 INFO: distutils: retargeting to non-venv dir '/opt/homebrew/Cellar/python@3.9/3.9.2_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9'
2367 INFO: Caching module dependency graph...
2410 INFO: running Analysis Analysis-00.toc
2413 INFO: Analyzing /Users/aciuffo/Documents/src/portfolioCreator/create_folders/createFolders.py
4635 INFO: Processing pre-find module path hook site from '/Users/aciuffo/.local/share/virtualenvs/portfolioCreator-ahiuVo4K/lib/python3.9/site-packages/PyInstaller/hooks/pre_find_module_path/hook-site.py'.
4635 INFO: site: retargeting to fake-dir '/Users/aciuffo/.local/share/virtualenvs/portfolioCreator-ahiuVo4K/lib/python3.9/site-packages/PyInstaller/fake-modules'
4920 INFO: Processing pre-safe import module hook six.moves from '/Users/aciuffo/.local/share/virtualenvs/portfolioCreator-ahiuVo4K/lib/python3.9/site-packages/PyInstaller/hooks/pre_safe_import_module/hook-six.moves.py'.
arch: posix_spawnp: /Users/aciuffo/.local/share/virtualenvs/portfolioCreator-ahiuVo4K/bin/python: Bad CPU type in executable
pre-safe-import-module hook failed, needs fixing.
```



Discussion:
This one's caused by some of our utility code still living in simpler times where `32-bit` == `i386` and `64-bit` == `x86_64`:
https://github.com/pyinstaller/pyinstaller/blob/227eac14955c02db21d4702429896d4b74beed5e/PyInstaller/compat.py#L489-L505. 

So we end up trying to run python with x86_64 arch, which does not work on M1 unless it happens to be `universal2` version (so it breaks on homebrew python, which is always single-arch). This in turn causes other utility functions such as `collect_submodules` in hooks come up empty...

If you need quick fix for that, you can try forcing `py_prefix` in the referenced code block to 
```python
py_prefix = ['arch', '-' + platform.machine()]
```
But do note that freezing on pure `arm64` does not work at the moment, unless you use the WIP branch that's trying to add support for that.
I think we probably ought to rename our bootloader directories whilst we're at it in that case. Linux is also now on `aarch64`, `ppc64le` and `s390x` - all of which are 64bit - with `aarch64` likely to become more popular thanks to the PineBook using it.

> I think we probably ought to rename our bootloader directories whilst we're at it in that case. Linux is also now on aarch64, ppc64le and s390x - all of which are 64bit - with aarch64 likely to become more popular thanks to the PineBook using it.

True, that should be cleaned up at some point as well. But it'll require changes in both bootloader's waf script and python-side code. I think currently, when compiling natively for those platforms on linux, the directories actually get named correctly. (But if you try to force arch manually, you hit [a check that prevents you from using anything else than '32bit' or '64bit'](https://github.com/pyinstaller/pyinstaller/blob/227eac14955c02db21d4702429896d4b74beed5e/bootloader/wscript#L261-L273)).

For macOS specifically, if we're going with `universal2` bootloader (which still allows us to extract single-arch slices during frozen application assembly) as currently planned, Darwin-64bit as bootloader directory actually works...
Fortunately code frozen on an i386 under Catalina still runs in Big Sur on both ARM and Intel hardware. I think for the moment I'll continue to build on i386 hardware as this particular program is used widely in our organization.

Is the scope of work to repair this such that I should plan to keep an i386/catalina machine for a long time? I only ask because I'll have to fight my IT department a bit to step out of the mandatory upgrades we roll out in July. 
I assume you mean x86_64 instead of i386? As far as I know, Catalina dropped support for 32-bit.

I personally think that at the moment, creating an x86_64 build is the safest way to ensure that your application will run everywhere. Because it will run on current Intel hardware, and on Apple M1 (with Rosetta translation). 

If you build using `arm64`-only python on M1, then the frozen application will obviously run only on M1, so you'll need to create an additional x86_64 build for Intel macs. (EDIT: that is, once we fully sort out support for M1)

We're planning to enable `universal2` builds of bootloader, which in theory could allow building `universal2` versions of frozen applications. But the problem is, this requires all used python packages to come with `universal2` libraries, otherwise the application won't run on one platform or the other (for example, `cryptography` package is currently providing distinct `x86_64` and `arm64` wheels; and homebrew python is completely single-arch anyway). 

So I think what we'll do is to build `universal2` bootloaders by default, then have them reduced to single-arch corresponding to current running architecture during freezing process, with an option to try forcing `universal2` build (and verifying that all libraries have required arch slices). This would also allow making either `universal2` frozen application, or separate single-arch builds using single `universal2` python environment (again, provided all packages are in `universal2` format).
So long story short, you'll probably want to keep x86_64 mac around, unless you're going to upgrade all your users to M1. And if the question was Catalina vs. Big Sur, the answer is kind of the same. If all your users will receive mandatory upgrade to Big Sur, then you can probably move your build environment to Big Sur as well. Otherwise, it might be safer to stick with Catalina unless you ensure that the versions of python packages installed in Big Sur build environments are also compatible with Catalina.
@rokm 
Thank you for the very clear explanation. Your thorough and complete advice is greatly appreciated. I can clearly see my path forward and understand what I need to do.

Resolving patch:
diff --git a/PyInstaller/compat.py b/PyInstaller/compat.py
--- a/PyInstaller/compat.py
+++ b/PyInstaller/compat.py
@@ -494,8 +494,15 @@ def __wrap_python(args, kwargs):
     # architecture as python executable.
     # It is necessary to run binaries with 'arch' command.
     if is_darwin:
-        mapping = {'32bit': '-i386', '64bit': '-x86_64'}
-        py_prefix = ['arch', mapping[architecture]]
+        if architecture == '64bit':
+            if machine == 'arm':
+                py_prefix = ['arch', '-arm64']  # Apple M1
+            else:
+                py_prefix = ['arch', '-x86_64']  # Intel
+        elif architecture == '32bit':
+            py_prefix = ['arch', '-i386']
+        else:
+            py_prefix = []
         # Since OS X 10.11 the environment variable DYLD_LIBRARY_PATH is no
         # more inherited by child processes, so we proactively propagate
         # the current value using the `-e` option of the `arch` command.


Test patch: