Repository: pyinstaller/pyinstaller

Issue:
PyInstaller 4.5.1 "codesign failure!" on macos
I'm trying to build my application on macOS and I'm getting this error with the latest PyInstaller 4.5.1, it is working on PyInstaller 4.3

```
[2021-08-26T07:28:02.721Z] 12064 WARNING: codesign command (['codesign', '-s', '-', '--force', '--all-architectures', '--timestamp', '/Users/safebreach/Library/Application Support/pyinstaller/bincache00_py37_64bit/x86_64/adhoc/no-entitlements/libintl.8.dylib']) failed with error code 1!
[2021-08-26T07:28:02.721Z] stdout: ''
[2021-08-26T07:28:02.721Z] stderr: '/Users/safebreach/Library/Application Support/pyinstaller/bincache00_py37_64bit/x86_64/adhoc/no-entitlements/libintl.8.dylib: invalid or unsupported format for signature\n'
[2021-08-26T07:28:02.721Z] Traceback (most recent call last):
[2021-08-26T07:28:02.721Z]   File "/Volumes/workspace/workspace/ConnectivityCheck_develop/connectivity_check_venv/bin/pyinstaller", line 8, in <module>
[2021-08-26T07:28:02.721Z]     sys.exit(run())
[2021-08-26T07:28:02.721Z]   File "/Volumes/workspace/workspace/ConnectivityCheck_develop/connectivity_check_venv/lib/python3.7/site-packages/PyInstaller/__main__.py", line 126, in run
[2021-08-26T07:28:02.721Z]     run_build(pyi_config, spec_file, **vars(args))
[2021-08-26T07:28:02.721Z]   File "/Volumes/workspace/workspace/ConnectivityCheck_develop/connectivity_check_venv/lib/python3.7/site-packages/PyInstaller/__main__.py", line 65, in run_build
[2021-08-26T07:28:02.721Z]     PyInstaller.building.build_main.main(pyi_config, spec_file, **kwargs)
[2021-08-26T07:28:02.721Z]   File "/Volumes/workspace/workspace/ConnectivityCheck_develop/connectivity_check_venv/lib/python3.7/site-packages/PyInstaller/building/build_main.py", line 815, in main
[2021-08-26T07:28:02.721Z]     build(specfile, kw.get('distpath'), kw.get('workpath'), kw.get('clean_build'))
[2021-08-26T07:28:02.721Z]   File "/Volumes/workspace/workspace/ConnectivityCheck_develop/connectivity_check_venv/lib/python3.7/site-packages/PyInstaller/building/build_main.py", line 762, in build
[2021-08-26T07:28:02.721Z]     exec(code, spec_namespace)
[2021-08-26T07:28:02.721Z]   File "cicd/connectivity_check.spec", line 55, in <module>
[2021-08-26T07:28:02.721Z]     console=True
[2021-08-26T07:28:02.721Z]   File "/Volumes/workspace/workspace/ConnectivityCheck_develop/connectivity_check_venv/lib/python3.7/site-packages/PyInstaller/building/api.py", line 515, in __init__
[2021-08-26T07:28:02.721Z]     entitlements_file=self.entitlements_file
[2021-08-26T07:28:02.721Z]   File "/Volumes/workspace/workspace/ConnectivityCheck_develop/connectivity_check_venv/lib/python3.7/site-packages/PyInstaller/building/api.py", line 208, in __init__
[2021-08-26T07:28:02.721Z]     self.__postinit__()
[2021-08-26T07:28:02.721Z]   File "/Volumes/workspace/workspace/ConnectivityCheck_develop/connectivity_check_venv/lib/python3.7/site-packages/PyInstaller/building/datastruct.py", line 159, in __postinit__
[2021-08-26T07:28:02.721Z]     self.assemble()
[2021-08-26T07:28:02.721Z]   File "/Volumes/workspace/workspace/ConnectivityCheck_develop/connectivity_check_venv/lib/python3.7/site-packages/PyInstaller/building/api.py", line 280, in assemble
[2021-08-26T07:28:02.721Z]     entitlements_file=self.entitlements_file)
[2021-08-26T07:28:02.721Z]   File "/Volumes/workspace/workspace/ConnectivityCheck_develop/connectivity_check_venv/lib/python3.7/site-packages/PyInstaller/building/utils.py", line 391, in checkCache
[2021-08-26T07:28:02.721Z]     osxutils.sign_binary(cachedfile, codesign_identity, entitlements_file)
[2021-08-26T07:28:02.721Z]   File "/Volumes/workspace/workspace/ConnectivityCheck_develop/connectivity_check_venv/lib/python3.7/site-packages/PyInstaller/utils/osx.py", line 383, in sign_binary
[2021-08-26T07:28:02.721Z]     raise SystemError("codesign failure!")
```

Is there a way to work around it?

PyInstaller: 4.5.1
Python: 3.7.9
OS: macOS



Discussion:
> [2021-08-26T07:28:02.721Z] stderr: '/Users/safebreach/Library/Application Support/pyinstaller/bincache00_py37_64bit/x86_64/adhoc/no-entitlements/libintl.8.dylib: invalid or unsupported format for signature\n'

Can you figure out where the offending `libintl.8.dylib` came from?

> Is there a way to work around it?

I suppose if you don't care about binaries not being code signed (even ad-hoc), you could comment the raise of `SystemError` in your local PyInstaller version ("PyInstaller/utils/osx.py", line 383, in sign_binary), so the error becomes non-fatal.
I have no idea what is this file, I have searched for it in the machine and this is what I have found

```
Build-Simulator-2:ConnectivityCheck_develop safebreach$ find / -name libintl.8.dylib 2>/dev/null
/usr/local/lib/libintl.8.dylib
/usr/local/Cellar/gettext/0.21/lib/libintl.8.dylib
/Users/safebreach/Library/Application Support/pyinstaller/bincache00_py37_64bit/libintl.8.dylib
/Applications/Wireshark.app/Contents/Frameworks/libintl.8.dylib
```

I really don't want to start editing the code to disable the codesign, why in 4.3 everything is working? 
> I really don't want to start editing the code to disable the codesign, why in 4.3 everything is working?

Because in 4.3, we do not (re-)sign the binaries we collect.
> I have no idea what is this file.

Can you attach the `Analysis-00.toc` file from your `build/<name>` directory?

Also, what macOS version are you using?
My build passes on my MacOS 11.5.2 machine but fails on my OSX 10.13.6 VM:
```
2444 WARNING: codesign command (['codesign', '-s', '-', '--force', '--all-architectures', '--timestamp', '/Users/lachlan/Library/Application Support/pyinstaller/bincache00_py39_64bit/x86_64/adhoc/no-entitlements/libreadline.8.dylib']) failed with error code 1!
stdout: ''
stderr: '/Users/lachlan/Library/Application Support/pyinstaller/bincache00_py39_64bit/x86_64/adhoc/no-entitlements/libreadline.8.dylib: invalid or unsupported format for signature\n'
Traceback (most recent call last):
  File "/Users/lachlan/autor1/env/bin/pyinstaller", line 8, in <module>
    sys.exit(run())
  File "/Users/lachlan/autor1/env/lib/python3.9/site-packages/PyInstaller/__main__.py", line 126, in run
    run_build(pyi_config, spec_file, **vars(args))
  File "/Users/lachlan/autor1/env/lib/python3.9/site-packages/PyInstaller/__main__.py", line 65, in run_build
    PyInstaller.building.build_main.main(pyi_config, spec_file, **kwargs)
  File "/Users/lachlan/autor1/env/lib/python3.9/site-packages/PyInstaller/building/build_main.py", line 815, in main
    build(specfile, kw.get('distpath'), kw.get('workpath'), kw.get('clean_build'))
  File "/Users/lachlan/autor1/env/lib/python3.9/site-packages/PyInstaller/building/build_main.py", line 762, in build
    exec(code, spec_namespace)
  File "/Users/lachlan/autor1/autor1.spec", line 23, in <module>
    exe = EXE(pyz,
  File "/Users/lachlan/autor1/env/lib/python3.9/site-packages/PyInstaller/building/api.py", line 509, in __init__
    self.pkg = PKG(self.toc, cdict=kwargs.get('cdict', None),
  File "/Users/lachlan/autor1/env/lib/python3.9/site-packages/PyInstaller/building/api.py", line 208, in __init__
    self.__postinit__()
  File "/Users/lachlan/autor1/env/lib/python3.9/site-packages/PyInstaller/building/datastruct.py", line 159, in __postinit__
    self.assemble()
  File "/Users/lachlan/autor1/env/lib/python3.9/site-packages/PyInstaller/building/api.py", line 274, in assemble
    fnm = checkCache(fnm, strip=self.strip_binaries,
  File "/Users/lachlan/autor1/env/lib/python3.9/site-packages/PyInstaller/building/utils.py", line 391, in checkCache
    osxutils.sign_binary(cachedfile, codesign_identity, entitlements_file)
  File "/Users/lachlan/autor1/env/lib/python3.9/site-packages/PyInstaller/utils/osx.py", line 383, in sign_binary
    raise SystemError("codesign failure!")
SystemError: codesign failure
```
And the same failure but for `pyinstaller/bincache00_py37_64bit/x86_64/adhoc/no-entitlements/lib-dynload/unicodedata.cpython-37m-darwin.so` under the current default (OSX 10.13.6) Travis environment.

Swapping from ad-hoc signing to self-signed passed as expected.
@lachyc How can I use "self-signed" instead of "ad-hoc" ?
> 
> 
> @lachyc How can I use "self-signed" instead of "ad-hoc" ?

@arossert 

1. Open `Keychain Access`

2. Go to the menu at the top of the screen and select `Keychain Access > Certificate Assistant > Create a Certificate`

3. In the window that appears change `Certificate Type` to `Code Signing` ,  select `Create` and note the name of the certificate

4. Build your `pyinstaller` source with the `--codesign-identity <name>` flag where `<name>` is the name of the certificate you created in step 3
> My build passes on my MacOS 11.5.2 machine but fails on my OSX 10.13.6 VM:

This does look like an issue with `codesign` on 10.13 - sometimes it seems to fail to properly remove the signature from a file, and then subsequent attempt to re-sign it fails... and this seems to happen only sometimes, even when given a fresh copy of the same source file (e.g., `lib-dynload/zlib.cpython-38-darwin.so`).

> Swapping from ad-hoc signing to self-signed passed as expected.

Does the self-signed app work correctly, though? If signing identity is provided, we turn on hardened runtime, and according to my tests, that did [not work with self-signed certificate](https://github.com/pyinstaller/pyinstaller/pull/5581#discussion_r611233365)...
> > Swapping from ad-hoc signing to self-signed passed as expected.
> 
> Does the self-signed app work correctly, though? If signing identity is provided, we turn on hardened runtime, and according to my tests, that did [not work with self-signed certificate](https://github.com/pyinstaller/pyinstaller/pull/5581#discussion_r611233365)...

Running pyinstaller 4.5.1 with a self-signed certificate on 10.13 generates an app that runs on both 10.13 and 11.5.2 for me.

Edit: Building on 11.5.2 with a self-signed cert works too.

I am generating a single binary rather than a .app bundle in case that makes any difference!

Resolving patch:
diff --git a/PyInstaller/building/utils.py b/PyInstaller/building/utils.py
--- a/PyInstaller/building/utils.py
+++ b/PyInstaller/building/utils.py
@@ -369,11 +369,13 @@ def checkCache(
     cache_index[basenm] = digest
     misc.save_py_data_struct(cacheindexfn, cache_index)
 
-    # On Mac OS we need relative paths to dll dependencies starting with @executable_path. Modifying headers
-    # invalidates signatures, so remove any existing signature and then re-add it after paths are rewritten.
+    # On Mac OS we need relative paths to dll dependencies starting with @executable_path. While modifying
+    # the headers invalidates existing signatures, we avoid removing them in order to speed things up (and
+    # to avoid potential bugs in the codesign utility, like the one reported on Mac OS 10.13 in #6167).
+    # The forced re-signing at the end should take care of the invalidated signatures.
     if is_darwin:
         osxutils.binary_to_target_arch(cachedfile, target_arch, display_name=fnm)
-        osxutils.remove_signature_from_binary(cachedfile)
+        #osxutils.remove_signature_from_binary(cachedfile)  # Disabled as per comment above.
         dylib.mac_set_relative_dylib_deps(cachedfile, dist_nm)
         osxutils.sign_binary(cachedfile, codesign_identity, entitlements_file)
     return cachedfile
diff --git a/PyInstaller/utils/osx.py b/PyInstaller/utils/osx.py
--- a/PyInstaller/utils/osx.py
+++ b/PyInstaller/utils/osx.py
@@ -186,18 +186,22 @@ def fix_exe_for_code_signing(filename):
     symtab_sec = [cmd for cmd in header.commands if cmd[0].cmd == LC_SYMTAB]
     assert len(symtab_sec) == 1, "Expected exactly one SYMTAB section!"
     symtab_sec = symtab_sec[0][1]  # Take the symtab command entry
-    # Sanity check; the string table is located at the end of the SYMTAB section, which in turn is the last section in
-    # the __LINKEDIT segment.
-    assert linkedit_seg.fileoff + linkedit_seg.filesize == symtab_sec.stroff + symtab_sec.strsize, \
-        "Sanity check failed!"
-
-    # Compute the old/declared file size (header.offset is zero for single-arch thin binaries).
-    old_file_size = header.offset + linkedit_seg.fileoff + linkedit_seg.filesize
-    delta = file_size - old_file_size
-    # Expand the string table in SYMTAB section...
-    symtab_sec.strsize += delta
-    # .. as well as its parent __LINEDIT segment.
-    linkedit_seg.filesize += delta
+
+    # The string table is located at the end of the SYMTAB section, which in turn is the last section in the __LINKEDIT
+    # segment. Therefore, the end of SYMTAB section should be aligned with the end of __LINKEDIT segment, and in turn
+    # both should be aligned with the end of the file (as we are in the last or the only arch slice).
+    #
+    # However, when removing the signature from the executable using codesign under Mac OS 10.13, the codesign utility
+    # may produce an invalid file, with the declared length of the __LINKEDIT segment (linkedit_seg.filesize) pointing
+    # beyond the end of file, as reported in issue #6167.
+    #
+    # We can compensate for that by not using the declared sizes anywhere, and simply recompute them. In the final
+    # binary, the __LINKEDIT segment and the SYMTAB section MUST end at the end of the file (otherwise, we have bigger
+    # issues...). So simply recompute the declared sizes as difference between the final file length and the
+    # corresponding start offset.
+    symtab_sec.strsize = file_size - symtab_sec.stroff
+    linkedit_seg.filesize = file_size - linkedit_seg.fileoff
+
     # Compute new vmsize by rounding filesize up to full page size.
     page_size = (0x4000 if _get_arch_string(header.header).startswith('arm64') else 0x1000)
     linkedit_seg.vmsize = math.ceil(linkedit_seg.filesize / page_size) * page_size


Test patch: