Repository: pyinstaller/pyinstaller

Issue:
Collect-all on Mac when DLLs exist tries to checkCache the DLLs.
## Description of the issue
running `pyinstaller --collect-all tkinterdnd2 -w sub_processor.py` on mac with a Conda installation collects the DLL binaries and tries to `PyInstaller.building.utils.checkCache` them. 
(For whatever reason, Conda/Pip install osx64, linux64, and win64 shared libraries).
This leads to line 303: 
`res = winmanifest.GetManifestResources(os.path.abspath(cachedfile))`
which is not imported on a mac for obvious reasons.

[The Docs](https://pyinstaller.readthedocs.io/en/stable/usage.html) only mention excluding *modules* not sub-directories.

The Error is resolved if I manually delete the win64 folder prior to running pyinstaller.

### Context information
* Output of `pyinstaller --version`: ```4.6```
* Version of Python: `Python 3.6.13 :: Anaconda, Inc.`
* Platform: `Mac OS 10.15.7`
* How you installed Python: `conda`
* Did you also try this on another platform? `NA`

### Make sure [everything is packaged correctly](https://github.com/pyinstaller/pyinstaller/wiki/How-to-Report-Bugs#make-sure-everything-is-packaged-correctly)

  * [ x] start with clean installation
  * [ x] use the latest development version
  * [x ] Run your frozen program **from a command window (shell)** — instead of double-clicking on it
  * [ x] Package your program in **--onedir mode**
  * [x ] Package **without UPX**, say: use the option `--noupx` or set `upx=False` in your .spec-file
  * [ x] Repackage you application in **verbose/debug mode**. For this, pass the option `--debug` to `pyi-makespec` or `pyinstaller` or use `EXE(..., debug=1, ...)` in your .spec file.


### A minimal example program which shows the error

```
# Obviously after having pip installed tkinterdnd2 0.3.0
from tkinterdnd2 import *
```

### Stacktrace / full error message


```
pyinstaller --onedir --noupx --collect-all tkinterdnd2 -w sub_processor.py
427 INFO: PyInstaller: 5.0.dev0
427 INFO: Python: 3.6.13 (conda)
437 INFO: Platform: Darwin-19.6.0-x86_64-i386-64bit
438 INFO: wrote /Volumes/documents/github/subtitle_proc/sub_processor.spec
443 INFO: UPX is not available.
599 INFO: Determining a mapping of distributions to packages...
4366 INFO: Packages required by tkinterdnd2:
[]
4366 INFO: Extending PYTHONPATH with paths
['/Volumes/documents/github/subtitle_proc']
4799 INFO: checking Analysis
4800 INFO: Building Analysis because Analysis-00.toc is non existent
4800 INFO: Initializing module dependency graph...
4803 INFO: Caching module graph hooks...
4822 INFO: Analyzing base_library.zip ...
9631 INFO: Caching module dependency graph...
9913 INFO: running Analysis Analysis-00.toc
9922 INFO: Analyzing /Volumes/documents/github/subtitle_proc/sub_processor.py
10438 INFO: Processing module hooks...
10439 INFO: Loading module hook 'hook-_tkinter.py' from '/Users/johngrey/anaconda/envs/subtitle/lib/python3.6/site-packages/PyInstaller/hooks'...
10725 INFO: checking Tree
10726 INFO: Building Tree because Tree-00.toc is non existent
10726 INFO: Building Tree Tree-00.toc
10752 INFO: checking Tree
10752 INFO: Building Tree because Tree-01.toc is non existent
10752 INFO: Building Tree Tree-01.toc
10882 INFO: checking Tree
10882 INFO: Building Tree because Tree-02.toc is non existent
10882 INFO: Building Tree Tree-02.toc
10886 INFO: Loading module hook 'hook-encodings.py' from '/Users/johngrey/anaconda/envs/subtitle/lib/python3.6/site-packages/PyInstaller/hooks'...
11043 INFO: Loading module hook 'hook-pickle.py' from '/Users/johngrey/anaconda/envs/subtitle/lib/python3.6/site-packages/PyInstaller/hooks'...
11048 INFO: Loading module hook 'hook-heapq.py' from '/Users/johngrey/anaconda/envs/subtitle/lib/python3.6/site-packages/PyInstaller/hooks'...
11051 INFO: Loading module hook 'hook-difflib.py' from '/Users/johngrey/anaconda/envs/subtitle/lib/python3.6/site-packages/PyInstaller/hooks'...
11053 INFO: Loading module hook 'hook-xml.py' from '/Users/johngrey/anaconda/envs/subtitle/lib/python3.6/site-packages/PyInstaller/hooks'...
11383 INFO: Looking for ctypes DLLs
11383 INFO: Analyzing run-time hooks ...
11386 INFO: Including run-time hook '/Users/johngrey/anaconda/envs/subtitle/lib/python3.6/site-packages/PyInstaller/hooks/rthooks/pyi_rth__tkinter.py'
11388 INFO: Including run-time hook '/Users/johngrey/anaconda/envs/subtitle/lib/python3.6/site-packages/PyInstaller/hooks/rthooks/pyi_rth_pkgutil.py'
11392 INFO: Including run-time hook '/Users/johngrey/anaconda/envs/subtitle/lib/python3.6/site-packages/PyInstaller/hooks/rthooks/pyi_rth_inspect.py'
11400 INFO: Looking for dynamic libraries
12085 INFO: Looking for eggs
12085 INFO: Python library not in binary dependencies. Doing additional searching...
12093 INFO: Using Python library /Users/johngrey/anaconda/envs/subtitle/lib/libpython3.6m.dylib
12102 INFO: Warnings written to /Volumes/documents/github/subtitle_proc/build/sub_processor/warn-sub_processor.txt
12162 INFO: Graph cross-reference written to /Volumes/documents/github/subtitle_proc/build/sub_processor/xref-sub_processor.html
12201 INFO: Appending 'binaries' from .spec
12202 INFO: Appending 'datas' from .spec
12209 INFO: checking PYZ
12209 INFO: Building PYZ because PYZ-00.toc is non existent
12209 INFO: Building PYZ (ZlibArchive) /Volumes/documents/github/subtitle_proc/build/sub_processor/PYZ-00.pyz
13028 INFO: Building PYZ (ZlibArchive) /Volumes/documents/github/subtitle_proc/build/sub_processor/PYZ-00.pyz completed successfully.
13039 INFO: EXE target arch: x86_64
13039 INFO: Code signing identity: None
13040 INFO: checking PKG
13040 INFO: Building PKG because PKG-00.toc is non existent
13041 INFO: Building PKG (CArchive) sub_processor.pkg
13063 INFO: Building PKG (CArchive) sub_processor.pkg completed successfully.
13066 INFO: Bootloader /Users/johngrey/anaconda/envs/subtitle/lib/python3.6/site-packages/PyInstaller/bootloader/Darwin-64bit/runw
13066 INFO: checking EXE
13066 INFO: Building EXE because EXE-00.toc is non existent
13066 INFO: Building EXE from EXE-00.toc
13066 INFO: Copying bootloader EXE to /Volumes/documents/github/subtitle_proc/build/sub_processor/sub_processor
13067 INFO: Converting EXE to target arch (x86_64)
13099 INFO: Removing signature(s) from EXE
13150 INFO: Appending PKG archive to EXE
13153 INFO: Fixing EXE headers for code signing
13185 INFO: Rewriting the executable's macOS SDK version (11.3.0) to match the SDK version of the Python library (10.10.0) in order to avoid inconsistent behavior and potential UI issues in the frozen application.
13192 INFO: Re-signing the EXE
13268 INFO: Building EXE from EXE-00.toc completed successfully.
13272 INFO: checking COLLECT
13273 INFO: Building COLLECT because COLLECT-00.toc is non existent
13273 INFO: Building COLLECT COLLECT-00.toc
Traceback (most recent call last):
  File "/Users/johngrey/anaconda/envs/subtitle/lib/python3.6/site-packages/PyInstaller/building/utils.py", line 320, in checkCache
    res = winmanifest.GetManifestResources(os.path.abspath(cachedfile))
NameError: name 'winmanifest' is not defined

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/johngrey/anaconda/envs/subtitle/bin/pyinstaller", line 8, in <module>
    sys.exit(run())
  File "/Users/johngrey/anaconda/envs/subtitle/lib/python3.6/site-packages/PyInstaller/__main__.py", line 124, in run
    run_build(pyi_config, spec_file, **vars(args))
  File "/Users/johngrey/anaconda/envs/subtitle/lib/python3.6/site-packages/PyInstaller/__main__.py", line 58, in run_build
    PyInstaller.building.build_main.main(pyi_config, spec_file, **kwargs)
  File "/Users/johngrey/anaconda/envs/subtitle/lib/python3.6/site-packages/PyInstaller/building/build_main.py", line 818, in main
    build(specfile, kw.get('distpath'), kw.get('workpath'), kw.get('clean_build'))
  File "/Users/johngrey/anaconda/envs/subtitle/lib/python3.6/site-packages/PyInstaller/building/build_main.py", line 750, in build
    exec(code, spec_namespace)
  File "/Volumes/documents/github/subtitle_proc/sub_processor.spec", line 56, in <module>
    name='sub_processor',
  File "/Users/johngrey/anaconda/envs/subtitle/lib/python3.6/site-packages/PyInstaller/building/api.py", line 829, in __init__
    self.__postinit__()
  File "/Users/johngrey/anaconda/envs/subtitle/lib/python3.6/site-packages/PyInstaller/building/datastruct.py", line 155, in __postinit__
    self.assemble()
  File "/Users/johngrey/anaconda/envs/subtitle/lib/python3.6/site-packages/PyInstaller/building/api.py", line 869, in assemble
    entitlements_file=self.entitlements_file
  File "/Users/johngrey/anaconda/envs/subtitle/lib/python3.6/site-packages/PyInstaller/building/utils.py", line 321, in checkCache
    except winresource.pywintypes.error as e:
NameError: name 'winresource' is not defined
make: *** [makefile:13: test] Error 1

```


Discussion:
Indeed, [winmanifest is only imported (and even installed) on Windows](https://github.com/pyinstaller/pyinstaller/blob/a4b6f0fefe1957c10b9abdbc2ba44daec0f2af58/PyInstaller/building/utils.py#L34-L35) but [the usage of winmanifest is guarded only with a check for Windows filetypes](https://github.com/pyinstaller/pyinstaller/blob/a4b6f0fefe1957c10b9abdbc2ba44daec0f2af58/PyInstaller/building/utils.py#L317-L320) so if a Windows DLL somehow gets into that function on a non Windows platform it falls to bits.

Whilst the above is a bug in its own right, I'm more uncomfortable with those DLLs being collected in the first place on non Windows platforms. I see that `collect_dynamic_libs()` (which is called by `collect_all()` which is invoked by the `--collect-all` flag) [indiscriminately collects binaries for all platforms](https://github.com/pyinstaller/pyinstaller/blob/a4b6f0fefe1957c10b9abdbc2ba44daec0f2af58/PyInstaller/utils/hooks/__init__.py#L693) which it really shouldn't do.

So two bugs in one. Yippee!
The Windows case should be easy to solve (I think) - [the guard for winmanifest usage](https://github.com/pyinstaller/pyinstaller/blob/a4b6f0fefe1957c10b9abdbc2ba44daec0f2af58/PyInstaller/building/utils.py#L317-L320) should be extended with `and (is_win or is_cygwin)`.

However, if a linux `.so` shared library ends up being collected on macOS, the binary processing code there will also fall apart (which is what also happens with `tkinterdnd2`, whose wheel ships binaries for all three platforms). In this case, we cannot do extension-based filtering, as `.so` is a valid shared library suffix on macOS... 

Resolving patch:
diff --git a/PyInstaller/building/utils.py b/PyInstaller/building/utils.py
--- a/PyInstaller/building/utils.py
+++ b/PyInstaller/building/utils.py
@@ -31,7 +31,7 @@
 from PyInstaller.depend.bindepend import match_binding_redirect
 from PyInstaller.utils import misc
 
-if is_win:
+if is_win or is_cygwin:
     from PyInstaller.utils.win32 import versioninfo, winmanifest, winresource
 
 if is_darwin:
@@ -245,7 +245,7 @@ def checkCache(
             return cachedfile
 
     # Optionally change manifest and its dependencies to private assemblies.
-    if fnm.lower().endswith(".manifest"):
+    if fnm.lower().endswith(".manifest") and (is_win or is_cygwin):
         manifest = winmanifest.Manifest()
         manifest.filename = fnm
         with open(fnm, "rb") as f:
@@ -276,7 +276,7 @@ def checkCache(
                 entitlements_file=entitlements_file
             )
         # We need to avoid using UPX with Windows DLLs that have Control Flow Guard enabled, as it breaks them.
-        if is_win and versioninfo.pefile_check_control_flow_guard(fnm):
+        if (is_win or is_cygwin) and versioninfo.pefile_check_control_flow_guard(fnm):
             logger.info('Disabling UPX for %s due to CFG!', fnm)
         elif misc.is_file_qt_plugin(fnm):
             logger.info('Disabling UPX for %s due to it being a Qt plugin!', fnm)
@@ -314,7 +314,7 @@ def checkCache(
             pass
     os.chmod(cachedfile, 0o755)
 
-    if os.path.splitext(fnm.lower())[1] in (".pyd", ".dll"):
+    if os.path.splitext(fnm.lower())[1] in (".pyd", ".dll") and (is_win or is_cygwin):
         # When shared assemblies are bundled into the app, they may optionally be changed into private assemblies.
         try:
             res = winmanifest.GetManifestResources(os.path.abspath(cachedfile))
@@ -374,10 +374,16 @@ def checkCache(
     # to avoid potential bugs in the codesign utility, like the one reported on Mac OS 10.13 in #6167).
     # The forced re-signing at the end should take care of the invalidated signatures.
     if is_darwin:
-        osxutils.binary_to_target_arch(cachedfile, target_arch, display_name=fnm)
-        #osxutils.remove_signature_from_binary(cachedfile)  # Disabled as per comment above.
-        dylib.mac_set_relative_dylib_deps(cachedfile, dist_nm)
-        osxutils.sign_binary(cachedfile, codesign_identity, entitlements_file)
+        try:
+            osxutils.binary_to_target_arch(cachedfile, target_arch, display_name=fnm)
+            #osxutils.remove_signature_from_binary(cachedfile)  # Disabled as per comment above.
+            dylib.mac_set_relative_dylib_deps(cachedfile, dist_nm)
+            osxutils.sign_binary(cachedfile, codesign_identity, entitlements_file)
+        except osxutils.InvalidBinaryError:
+            # Raised by osxutils.binary_to_target_arch when the given file is not a valid macOS binary (for example,
+            # a linux .so file; see issue #6327). The error prevents any further processing, so just ignore it.
+            pass
+
     return cachedfile
 
 
diff --git a/PyInstaller/utils/osx.py b/PyInstaller/utils/osx.py
--- a/PyInstaller/utils/osx.py
+++ b/PyInstaller/utils/osx.py
@@ -261,12 +261,22 @@ def _get_arch_string(header):
     assert False, 'Unhandled architecture!'
 
 
+class InvalidBinaryError(Exception):
+    """
+    Exception raised by ˙get_binary_architectures˙ when it is passed an invalid binary.
+    """
+    pass
+
+
 def get_binary_architectures(filename):
     """
     Inspects the given binary and returns tuple (is_fat, archs), where is_fat is boolean indicating fat/thin binary,
     and arch is list of architectures with lipo/codesign compatible names.
     """
-    executable = MachO(filename)
+    try:
+        executable = MachO(filename)
+    except ValueError as e:
+        raise InvalidBinaryError("Invalid Mach-O binary!") from e
     return bool(executable.fat), [_get_arch_string(hdr.header) for hdr in executable.headers]
 
 


Test patch: