Repository: pyinstaller/pyinstaller

Issue:
genpy hook breaks win32com.client.gencache.EnsureDispatch()
## Description of the issue
The [runtime hook for genpy](https://github.com/pyinstaller/pyinstaller/blob/develop/PyInstaller/hooks/rthooks/pyi_rth_win32comgenpy.py) creates a temporary directory for the genpy cache and overwrites `win32com.__gen_path__` accordingly: https://github.com/pyinstaller/pyinstaller/blob/14c53a9d9f7b9322cfc8e18ae1c6e415230fba22/PyInstaller/hooks/rthooks/pyi_rth_win32comgenpy.py#L42
This creates a new cache such as:
C:\\Users\\<USER>\\AppData\\Local\\Temp\\**tmpytt5_e6s**\\gen_py (let's call this **hook cache**)

If a genpy cache already exists in:
C:\\Users\\<USER>\\AppData\\Local\\Temp\\gen_py (let's call this one **global cache**)
(as caused by running the python script directly, instead of the pyinstaller version)
the global cache is found by `EnsureDispatch()` such that no new cache will be generated in the hook cache.

This should be fine as long as the global cache contains all required python modules. However, if it does not, the win32com package tries to generate the missing modules:
This leads to the first exception shown below: `ModuleNotFoundError`, which is expected behaviour as far as I understand this mechanism.

But now, when handling this exception, since the hook has overwritten `win32com.__gen_path__`, generating the missing modules is attempted in the hook cache although the global cache is being used currently.
This leads to the second exception shown below: `No such file or directory`.

I'm not sure about the correct way to fix this. I can think of at least the following two ways:
1. Prevent the win32com package from discovering the global cache such that an entirely new cache is generated each time.
2. Modify the runtime hook such that the missing modules are generated in the currently used cache (i.e. the global cache if one exists and the hook cache if not global cache was found).

### Context information (for bug reports)

* Output of `pyinstaller --version`: ```5.0.dev0```
* Version of Python: 3.9.2
* Platform: Windows
* How you installed Python: python.org/downloads
* Did you also try this on another platform? Does it work there? No, since this is only applicable to Windows.

### A minimal example program which shows the error

```
from win32com.client import gencache

excel = gencache.EnsureDispatch("Excel.Application")

if not input(">"):
    # this will generate a cache that is complete for this "if" branch, but incomplete for the "else" branch.
    print(excel.Selection)
else:
    print(excel.Selection) 
    print(excel.Selection.Font)

```

To reproduce the error:
1. Open any excel sheet
2. Make sure that no global cache exists (simply delete it, if it exists)
3. Create the executable using: `pyinstaller bug_demo.py`
4. Run `bug_demo.exe`. It will prompt for an input. This should work no matter if you take the "if" branch (simply press Enter) or the "else" branch (provide some actual input, e.g. "a").
5. Run `bug_demo.py` directly using the python interpreter and just press Enter when it asks for input. This will create a global cache, which is incomplete for the "else" branch.
6.  Run `bug_demo.exe` again. This time, it will only work if you take the "if" branch (simply press Enter). If you take the "else" branch (provide some actual input, e.g. "a"), the error shown below should occur.

### Stacktrace / full error message


```
(venv) PS C:\Users\<SNIP>> .\dist\bug_demo\bug_demo.exe
a
None
Traceback (most recent call last):
  File "win32com\client\gencache.py", line 233, in GetModuleForCLSID
ModuleNotFoundError: No module named 'win32com.gen_py.00020813-0000-0000-C000-000000000046x0x1x9.Font'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "bug_demo.py", line 9, in <module>
    print(excel.Selection.Font)
  File "win32com\client\__init__.py", line 474, in __getattr__
  File "win32com\client\__init__.py", line 466, in _ApplyTypes_
  File "win32com\client\__init__.py", line 486, in _get_good_object_
  File "win32com\client\__init__.py", line 502, in _get_good_object_
  File "win32com\client\__init__.py", line 491, in _get_good_single_object_
  File "win32com\client\__init__.py", line 96, in Dispatch
  File "win32com\client\__init__.py", line 37, in __WrapDispatch
  File "win32com\client\gencache.py", line 180, in GetClassForCLSID
  File "win32com\client\gencache.py", line 241, in GetModuleForCLSID
  File "win32com\client\makepy.py", line 319, in GenerateChildFromTypeLibSpec
  File "win32com\client\genpy.py", line 1042, in generate_child
  File "win32com\client\genpy.py", line 779, in open_writer
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\<USER>\\AppData\\Local\\Temp\\tmpytt5_e6s\\gen_py\\00020813-0000-0000-C000-000000000046x0x1x9\\Font.py.18792.temp'
[18792] Failed to execute script bug_demo

```



Discussion:
> If a genpy cache already exists in:
C:\Users\\AppData\Local\Temp\gen_py (let's call this one global cache)
(as caused by running the python script directly, instead of the pyinstaller version)

You seem to be implying that the non-PyInstaller behaviour is to put that cache in a temporary folder instead of at `lib\sitepackages\win32com\client\gen_py.1?` In which case the runtime hook, whose job it was to trick win32com into putting those files in a writeable temporary folder, is now redundant?
The way I understand the current behaviour of win32com, it is to put a newly created cache into a temporary folder by default, yes:
https://github.com/mhammond/pywin32/blob/2a4b8dafe7d481a466077f7380f06b32e0db61a2/com/win32com/__init__.py#L104-L114

It does search for existing caches in `lib\site-packages\win32com\gen_py`, however.

Since separate caches are created for different python versions, it might be safe to remove the hook entirely, yes. That way, even pyinstaller versions can make use of an existing cache.

Resolving patch:
diff --git a/PyInstaller/hooks/rthooks/pyi_rth_win32comgenpy.py b/PyInstaller/hooks/rthooks/pyi_rth_win32comgenpy.py
--- a/PyInstaller/hooks/rthooks/pyi_rth_win32comgenpy.py
+++ b/PyInstaller/hooks/rthooks/pyi_rth_win32comgenpy.py
@@ -9,12 +9,12 @@
 # SPDX-License-Identifier: Apache-2.0
 #-----------------------------------------------------------------------------
 
-# The win32.client.gencache code must be allowed to create the cache in %temp% (user's temp). It is necessary to get the
-# gencache code to use a suitable directory other than the default in lib\site-packages\win32com\client\gen_py.
-# PyInstaller does not provide this directory structure and the frozen executable could be placed in a non-writable
-# directory like 'C:\Program Files. That's the reason for %temp% directory.
-#
-# http://www.py2exe.org/index.cgi/UsingEnsureDispatch
+# Put the cache generated by `win32com.client.gencache` into isolated temporary directory. Historically, this was
+# required due to earlier versions of `pywin32` using the `site-packages\win32com\client\gen_py`  directory for
+# the cache by default. Nowadays, the default location for the cache seems to be in the configured temporary directory
+# (pointed to by TEMP or TMP, for example %LOCALAPPDATA%\Temp), so strictly speaking, the relocation is not necessary
+# anymore. But for the time being, we are keeping it around to isolate the frozen application from the rest of the
+# system.
 
 
 def _pyi_rthook():
@@ -23,36 +23,22 @@ def _pyi_rthook():
     import shutil
     import tempfile
 
-    # Put gen_py cache in temp directory.
+    import win32com
+
+    # Create temporary directory. The actual cache directory needs to be named `gen_py`, so create a sub-directory.
     supportdir = tempfile.mkdtemp()
-    # gen_py has to be put into directory 'gen_py'.
+
     genpydir = os.path.join(supportdir, 'gen_py')
+    os.makedirs(genpydir, exist_ok=True)
 
-    # Create 'gen_py' directory. This directory does not need to contain '__init__.py' file.
-    try:
-        # win32com gencache cannot be put directly to 'supportdir' with any random name. It has to be put in a directory
-        # called 'gen_py'. This is the reason why to create this directory in supportdir'.
-        os.makedirs(genpydir)
-        # Remove temp directory at application exit and ignore any errors.
-        atexit.register(shutil.rmtree, supportdir, ignore_errors=True)
-    except OSError:
-        pass
+    # Remove the teporary directory at application exit, ignoring errors.
+    atexit.register(shutil.rmtree, supportdir, ignore_errors=True)
 
     # Override the default path to gen_py cache.
-    import win32com  # noqa: E402
-
     win32com.__gen_path__ = genpydir
 
-    # The attribute __loader__ makes module 'pkg_resources' working but On Windows it breaks pywin32 (win32com) and test
-    # 'basic/test_pyttsx' will fail. Just removing that attribute for win32com fixes that and gencache is created
-    # properly.
-    if hasattr(win32com, '__loader__'):
-        del win32com.__loader__
-
-    # Ensure genpydir is in 'gen_py' module paths.
-    import win32com.gen_py  # noqa: E402
-
-    win32com.gen_py.__path__.insert(0, genpydir)
+    # Override the sub-module paths for win32com.gen_py run-time sub-package.
+    win32com.gen_py.__path__ = [genpydir]
 
 
 _pyi_rthook()


Test patch: