Repository: pyinstaller/pyinstaller

Issue:
tkinter GUI's window is black
env:
```
MacOS Big Sur 11.3（20E232）
Python 3.9.2
pyinstaller 4.3
tkinter 8.6
```

I write a simply python application to test tkinter and pyinstaller.

I can run the python code (`python main.py`) on my MacOS.
But  I use pyinstaller to build application (`pyinstaller -w main.py`), the application windows is FULL BLACK.

I found the same question on [stackOverflow](https://stackoverflow.com/questions/35999344/how-to-determine-what-version-of-python3-tkinter-is-installed-on-my-linux-machin), but the question is unresolved.

And I try few way to fix this question,
1.  fix Tcl version `package require -exact Tcl 8.5.9` in `init.tcl` 
2. copy tcl/tkinter file from `/Library/Frameworks/Python.framework/Versions/3.9/lib` to `dist/main.app/Contents/Resources`
But they doesn't work


pyinstaller message:
```
% pyinstaller -w main.py 
46 INFO: PyInstaller: 4.3
46 INFO: Python: 3.9.2
73 INFO: Platform: macOS-10.16-x86_64-i386-64bit
74 INFO: wrote /Users/ted/Documents/workspace/knock/main.spec
78 INFO: UPX is not available.
80 INFO: Extending PYTHONPATH with paths
['/Users/ted/Documents/workspace', '/Users/ted/Documents/workspace/knock']
103 INFO: checking Analysis
104 INFO: Building Analysis because Analysis-00.toc is non existent
104 INFO: Initializing module dependency graph...
108 INFO: Caching module graph hooks...
114 WARNING: Several hooks defined for module 'win32ctypes.core'. Please take care they do not conflict.
120 INFO: Analyzing base_library.zip ...
2890 INFO: Processing pre-find module path hook distutils from '/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/PyInstaller/hooks/pre_find_module_path/hook-distutils.py'.
2896 INFO: distutils: retargeting to non-venv dir '/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9'
5274 INFO: Caching module dependency graph...
5426 INFO: running Analysis Analysis-00.toc
5442 INFO: Analyzing /Users/ted/Documents/workspace/knock/main.py
5445 INFO: Processing module hooks...
5445 INFO: Loading module hook 'hook-xml.etree.cElementTree.py' from '/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/PyInstaller/hooks'...
5448 INFO: Loading module hook 'hook-lib2to3.py' from '/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/PyInstaller/hooks'...
5494 INFO: Loading module hook 'hook-_tkinter.py' from '/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/PyInstaller/hooks'...
5638 INFO: checking Tree
5638 INFO: Building Tree because Tree-00.toc is non existent
5638 INFO: Building Tree Tree-00.toc
5656 INFO: checking Tree
5657 INFO: Building Tree because Tree-01.toc is non existent
5657 INFO: Building Tree Tree-01.toc
5733 INFO: checking Tree
5734 INFO: Building Tree because Tree-02.toc is non existent
5734 INFO: Building Tree Tree-02.toc
5738 INFO: Loading module hook 'hook-encodings.py' from '/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/PyInstaller/hooks'...
5806 INFO: Loading module hook 'hook-distutils.util.py' from '/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/PyInstaller/hooks'...
5810 INFO: Loading module hook 'hook-pickle.py' from '/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/PyInstaller/hooks'...
5813 INFO: Loading module hook 'hook-heapq.py' from '/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/PyInstaller/hooks'...
5814 INFO: Loading module hook 'hook-difflib.py' from '/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/PyInstaller/hooks'...
5816 INFO: Loading module hook 'hook-multiprocessing.util.py' from '/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/PyInstaller/hooks'...
5818 INFO: Loading module hook 'hook-sysconfig.py' from '/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/PyInstaller/hooks'...
5829 INFO: Loading module hook 'hook-xml.py' from '/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/PyInstaller/hooks'...
5884 INFO: Loading module hook 'hook-distutils.py' from '/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/PyInstaller/hooks'...
5898 INFO: Looking for ctypes DLLs
5923 INFO: Analyzing run-time hooks ...
5926 INFO: Including run-time hook '/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/PyInstaller/hooks/rthooks/pyi_rth_multiprocessing.py'
5930 INFO: Including run-time hook '/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/PyInstaller/hooks/rthooks/pyi_rth__tkinter.py'
5936 INFO: Looking for dynamic libraries
6064 INFO: Looking for eggs
6064 INFO: Using Python library /Library/Frameworks/Python.framework/Versions/3.9/Python
6067 INFO: Warnings written to /Users/ted/Documents/workspace/knock/build/main/warn-main.txt
6089 INFO: Graph cross-reference written to /Users/ted/Documents/workspace/knock/build/main/xref-main.html
6109 INFO: checking PYZ
6109 INFO: Building PYZ because PYZ-00.toc is non existent
6109 INFO: Building PYZ (ZlibArchive) /Users/ted/Documents/workspace/knock/build/main/PYZ-00.pyz
6475 INFO: Building PYZ (ZlibArchive) /Users/ted/Documents/workspace/knock/build/main/PYZ-00.pyz completed successfully.
6481 INFO: checking PKG
6481 INFO: Building PKG because PKG-00.toc is non existent
6481 INFO: Building PKG (CArchive) PKG-00.pkg
6496 INFO: Building PKG (CArchive) PKG-00.pkg completed successfully.
6497 INFO: Bootloader /Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/PyInstaller/bootloader/Darwin-64bit/runw
6497 INFO: checking EXE
6497 INFO: Building EXE because EXE-00.toc is non existent
6497 INFO: Building EXE from EXE-00.toc
6497 INFO: Appending archive to EXE /Users/ted/Documents/workspace/knock/build/main/main
6500 INFO: Fixing EXE for code signing /Users/ted/Documents/workspace/knock/build/main/main
6506 INFO: Building EXE from EXE-00.toc completed successfully.
6507 INFO: checking COLLECT
6508 INFO: Building COLLECT because COLLECT-00.toc is non existent
6508 INFO: Building COLLECT COLLECT-00.toc
7725 INFO: Building COLLECT COLLECT-00.toc completed successfully.
7741 INFO: checking BUNDLE
7742 INFO: Building BUNDLE because BUNDLE-00.toc is non existent
7742 INFO: Building BUNDLE BUNDLE-00.toc
8831 INFO: moving BUNDLE data files to Resource directory
```

code:
```
import tkinter as tk
import os
os.environ["QT_MAC_WANTS_LAYER"] = "1"

class Application(tk.Frame):
    def __init__(self, master=None):
        super().__init__(master)
        self.master = master
        self.pack()
        self.create_widgets()

    def create_widgets(self):
        self.hi_there = tk.Button(self)
        self.hi_there["text"] = "Hello World\n(click me)"
        self.hi_there["command"] = self.say_hi
        self.hi_there.pack(side="top")

        self.quit = tk.Button(self, text="QUIT", fg="red",
                              command=self.master.destroy)
        self.quit.pack(side="bottom")

    def say_hi(self):
        print("hi there, everyone!")
        

root = tk.Tk()
app = Application(master=root)
app.mainloop()
```




Discussion:
Try using the "universal2"/"macos11.0" version of python.

This seems to be caused  by mismatch in SDK versions between the bootloader and the Python library. Using PyInstaller 4.3 (which has bootloader built with XCode 12.x and macOS SDK 11.x) works with "universal2"/"macos11.0" version of python, while using it with "Intel"/"macos10.9" version does not (black screen). Using PyInstaller 4.2 (which has bootloader built with old version of XCode) works with both versions of python (with some limitations on Big Sur)...
Hmm, so it seems that this issue is complicated enough that trying to solve it will bring us to the wastelands of diminishing returns...

The actual problem here might not be the SDK version per-se, but rather the version of Tcl/Tk that's used by python.org builds (or at least, interaction between the two). Tcl/Tk 8.6.8 is used by `Intel` python.org installers, whereas the `universal2`/`macos11` installer switched to 8.6.11. That's why the work-around is to either use `universal2` version of python 3.9.x, or at least copy its Tcl and Tk libraries over to older python versions (a work-around for [a similar issue with cx_freeze](https://github.com/marcelotduarte/cx_Freeze/issues/849)).

The issue does go away if bootloader is built with older SDK version (as it was in PyInstaller 4.2 and older). But that's not feasible going forward, because it precludes proper macOS 11 support, so it is just trading one sort of problems for another (inconsistent behavior with reported version, with `ctypes.util.find_library()`, etc.).

Now, we could modify our bootloader build script to detect macOS SDK version, and use different install path for bootloaders built with macOS SDK 11.x and bootloaders built with older SDK. And we could modify our assembly pipeline to check for macOS SDK version against which the Python library (and hopefully by extension, Tcl and Tk libraries) was built, and then select the bootloader variant accordingly. Here's [proof-of-concept](https://github.com/rokm/pyinstaller/tree/macos_sdk_11_to_darwin20).

However, the problem is that in order for those bundled Tcl/Tk 8.6.8 to work properly, we need to build with really old SDK version. 10.15 (which is still available in Xcode Command Line Tools 12.4) doesn't work, and neither does 10.14 from Xcode 10.3. It does work with SDK 10.9 extracted from Xcode 6.4, but that sets sdk version in binary to "n/a" when used with compiler from Xcode Command Line Tools 12.x... (but it should work correctly when built with cross-compiling toolchain, as was done with bootloaders in earlier PyInstaller releases).

This means that whenever someone rebuilt bootloaders on macOS with recent version of Xcode, they ended up with version that exhibited this broken `tkinter` behavior...

Considering the above, maintaining two variants of prebuilt bootloaders becomes a real hassle, as it requires two toolchains (recent and old XCode, or two cross-compile toolchains, one for each SDK version). It also complicates the building of bootloader on macOS itself, because by default, one version will be (re)built, and that may not be the version that PyInstaller will actually be picking up (as the former depends on the Xcode version/settings, and the latter depends on what python is used). And I'm not sure we can properly set it up rebuilding of both variants on the CI, either.
Looks like the oldest we can manage on CI is [Xcode 8.0 on macOS 10.13](https://github.com/actions/virtual-environments/blob/main/images/macos/macos-10.13-Readme.md#xcode). Is that far back enough? That said, even if it is, it probably won't be available for much longer so it'd just be delaying the problem.

It sounds like we're just going to have to build several bootloaders. An extreme approach: If we went down the same path as Python extension modules of building a separate executable for every minor Python version which each mirrored the SDK/macOS deployment target of the Python version they are built for, would that resolve it? Granted we'd have to forget cross compilation and move everything to CI...
> Looks like the oldest we can manage on CI is Xcode 8.0 on macOS 10.13. Is that far back enough? That said, even if it is, it probably won't be available for much longer so it'd just be delaying the problem.

Hmm, that could be old enough (actually, Xcode 8.0 seems to come with 10.12 SDK). I've noticed that our older bootloaders were built against 10.11 (Xcode 7.3.1), not 10.9. And I suspect that the first problematic SDK is actually 10.14, as Mojave seems to be the macOS version that began overhauling the UI...

But I think we only need just two targets, a 10.9 one (or rather, 10.11, which will be equivalent to what we had in 4.2 and earlier) and an 11.x one (for proper Big Sur support). These also match the SDK versions against which python.org installers are built. I'm not sure how this plays out on Homebrew and Anaconda python, though. 

> Granted we'd have to forget cross compilation and move everything to CI...

Actually, our cross-compiling VMs are the only environments that are fully controllable and that can ensure the desired results. For the version with old SDK, we have existing Vagrant, and for 11.x, we have Vagrant in the M1 PR (or the Docker file, which I presume was used for 4.3; but we could probably turn the 10.11 Vagrant VM into Docker as well).

What worries me is that compiling natively for 10.11 will be rather difficult. And same goes for the CI. Because in both cases if a recent toolchain with recent SDK is used by default (which I think is likely), it will compile the 11.x bootloader variant, regardless of python version and of OS version. Which will ensue in confusion if the python was built against 10.x SDK, because the other, unmodified variant will be picked up by PyInstaller.

Also, the wheel builder will need to be expanded to allow for more than one bootloader directory per platform. The macOS wheel needs to include all/both bootloader variants, because the required variant cannot be inferred in advance from OS version nor python version (python 3.9.x Intel installer is still build against 11.9, while universal2/macos11 installer is built against 11.x).
I can live with two bootloaders, even if we do need to restructure a bit to accommodate them.

As for Anaconda, the [conda-forge PyInstaller repo](https://github.com/conda-forge/pyinstaller-feedstock/) rebuild the bootloaders themselves. They build them per-Python version but use our waf script to do it (so they've currently got 4 identical Conda packages for each platform). In which case they'll also be hit by this. Probably the best path for them would be to forward Python's macOS deployment target into waf somehow seeing as they already build separate bootloaders per Python version.

Sounds rather defeatist but is there really much of a use case for compiling the bootloader natively on macOS?
Hmmm... actually, there might be an alternative... *solution*. 

The root of the problem here seems to be that Tk 8.6.8 that's used by python.org builds does not play nicely with dark mode that was first introduced in Mojave ([ref1](https://stackoverflow.com/questions/55483507/how-to-get-a-black-file-dialog-box-using-tkinter-on-mac-os/55516222#55516222), [ref2](https://stackoverflow.com/questions/52929744/python-tkinter-not-support-macos-mojave-dark-mode), [ref3](https://core.tcl-lang.org/tk/tktview?name=821dbe47e1)). 

So how does macOS decide whether to enable the dark mode for a program or not? Unsurprisingly, it has little to go on, and what it seems to be doing is checking the `sdk` field in `LC_VERSION_MIN_MACOSX` load command in the program's header. (And *that* is why toolchain/SDK used to compile the bootloader seems to matter here...).

So we could check if `sdk` in `LC_VERSION_MIN_MACOSX` in Python library's header is 10.9 and if it is, assume we're dealing with Intel python.org build, and rewrite the value in the assembled binary's header to 10.11 to match that of the old bootloaders.
Go for it!

Resolving patch:
diff --git a/PyInstaller/building/api.py b/PyInstaller/building/api.py
--- a/PyInstaller/building/api.py
+++ b/PyInstaller/building/api.py
@@ -635,9 +635,39 @@ def assemble(self):
                     shutil.copyfileobj(infh, outf, length=64*1024)
 
         if is_darwin:
+            import PyInstaller.utils.osx as osxutils
+
+            # If the version of macOS SDK used to build bootloader exceeds
+            # that of macOS SDK used to built Python library (and, by
+            # extension, bundled Tcl/Tk libraries), force the version
+            # declared by the frozen executable to match that of the Python
+            # library.
+            # Having macOS attempt to enable new features (based on SDK
+            # version) for frozen application has no benefit if the Python
+            # library does not support them as well.
+            # On the other hand, there seem to be UI issues in tkinter
+            # due to failed or partial enablement of dark mode (i.e., the
+            # bootloader executable being built against SDK 10.14 or later,
+            # which causes macOS to enable dark mode, and Tk libraries being
+            # built against an earlier SDK version that does not support the
+            # dark mode). With python.org Intel macOS installers, this
+            # manifests as black Tk windows and UI elements (see issue #5827),
+            # while in Anaconda python, it may result in white text on bright
+            # background.
+            pylib_version = osxutils.get_macos_sdk_version(
+                bindepend.get_python_library_path())
+            exe_version = osxutils.get_macos_sdk_version(self.name)
+            if pylib_version < exe_version:
+                logger.info(
+                    "Rewriting executable's macOS SDK version (%d.%d.%d) to "
+                    "match the SDK version of the Python library (%d.%d.%d) "
+                    "in order to avoid inconsistent behavior and potential UI "
+                    "issues in the frozen application.", *exe_version,
+                    *pylib_version)
+                osxutils.set_macos_sdk_version(self.name, *pylib_version)
+
             # Fix Mach-O header for codesigning on OS X.
             logger.info("Fixing EXE for code signing %s", self.name)
-            import PyInstaller.utils.osx as osxutils
             osxutils.fix_exe_for_code_signing(self.name)
         if is_win:
             # Set checksum to appease antiviral software.
diff --git a/PyInstaller/utils/osx.py b/PyInstaller/utils/osx.py
--- a/PyInstaller/utils/osx.py
+++ b/PyInstaller/utils/osx.py
@@ -19,6 +19,7 @@
 
 from PyInstaller.compat import base_prefix
 from macholib.MachO import MachO
+from macholib.mach_o import LC_BUILD_VERSION, LC_VERSION_MIN_MACOSX
 
 
 def is_homebrew_env():
@@ -67,6 +68,65 @@ def get_macports_prefix():
     return prefix
 
 
+def _find_version_cmd(header):
+    """
+    Helper that finds the version command in the given MachO header.
+    """
+    # The SDK version is stored in LC_BUILD_VERSION command (used when
+    # targeting the latest versions of macOS) or in older LC_VERSION_MIN_MACOSX
+    # command. Check for presence of either.
+    version_cmd = [cmd for cmd in header.commands
+                   if cmd[0].cmd in {LC_BUILD_VERSION, LC_VERSION_MIN_MACOSX}]
+    assert len(version_cmd) == 1, \
+        "Expected exactly one LC_BUILD_VERSION or " \
+        "LC_VERSION_MIN_MACOSX command!"
+    return version_cmd[0]
+
+
+def get_macos_sdk_version(filename):
+    """
+    Obtain the version of macOS SDK against which the given binary
+    was built.
+
+    NOTE: currently, version is retrieved only from the first arch
+    slice in the binary.
+
+    :return: (major, minor, revision) tuple
+    """
+    binary = MachO(filename)
+    header = binary.headers[0]
+    # Find version command using helper
+    version_cmd = _find_version_cmd(header)
+    # Parse SDK version number
+    major = (version_cmd[1].sdk & 0xFF0000) >> 16
+    minor = (version_cmd[1].sdk & 0xFF00) >> 8
+    revision = (version_cmd[1].sdk & 0xFF)
+    return major, minor, revision
+
+
+def set_macos_sdk_version(filename, major, minor, revision):
+    """
+    Overwrite the macOS SDK version declared in the given binary with
+    the specified version.
+
+    NOTE: currently, only version in the first arch slice is modified.
+    """
+    # Validate values
+    assert major >= 0 and major <= 255, "Invalid major version value!"
+    assert minor >= 0 and minor <= 255, "Invalid minor version value!"
+    assert revision >= 0 and revision <= 255, "Invalid revision value!"
+    # Open binary
+    binary = MachO(filename)
+    header = binary.headers[0]
+    # Find version command using helper
+    version_cmd = _find_version_cmd(header)
+    # Write new SDK version number
+    version_cmd[1].sdk = major << 16 | minor << 8 | revision
+    # Write changes back.
+    with open(binary.filename, 'rb+') as fp:
+        binary.write(fp)
+
+
 def fix_exe_for_code_signing(filename):
     """
     Fixes the Mach-O headers to make code signing possible.


Test patch: