Repository: nvaccess/nvda

Issue:
Current master: SCons source faills for the first two runs
### Steps to reproduce:
1. Clone fresh NVDA repo passing --recursive option as described in the readme.
2. Execute scons source from the root of the repo.
### Actual behavior:
Output of first scons run ends with the following:
scons: *** [source\comInterfaces\SpeechLib.py] ModuleNotFoundError : No module named 'comtypes.gen._C866CA3A_32F7_11D2_9602_00C04F8EE628_0_5_4'
Traceback (most recent call last):
  File "D:\my_repos\test\nvda\include\scons\src\engine\SCons\Action.py", line 1202, in execute
    result = self.execfunction(target=target, source=rsources, env=env)
  File "D:\my_repos\test\nvda\source\comInterfaces_sconscript", line 27, in interfaceAction
    comtypes.client.GetModule(source)
  File "D:\my_repos\test\nvda\include\comtypes\comtypes\client\_generate.py", line 110, in GetModule
    mod = _CreateWrapper(tlib, pathname)
  File "D:\my_repos\test\nvda\include\comtypes\comtypes\client\_generate.py", line 184, in _CreateWrapper
    mod = _my_import(fullname)
  File "D:\my_repos\test\nvda\include\comtypes\comtypes\client\_generate.py", line 24, in _my_import
    return __import__(fullname, globals(), locals(), ['DUMMY'])
ModuleNotFoundError: No module named 'comtypes.gen._C866CA3A_32F7_11D2_9602_00C04F8EE628_0_5_4'
scons: building terminated because of errors.

Output of second scons run ends with the following:

scons: *** [source\comInterfaces\tom.py] ModuleNotFoundError : No module named 'comtypes.gen._8CC497C9_A1DF_11CE_8098_00AA0047BE5D_0_1_0'
Traceback (most recent call last):
  File "D:\my_repos\test\nvda\include\scons\src\engine\SCons\Action.py", line 1202, in execute
    result = self.execfunction(target=target, source=rsources, env=env)
  File "D:\my_repos\test\nvda\source\comInterfaces_sconscript", line 27, in interfaceAction
    comtypes.client.GetModule(source)
  File "D:\my_repos\test\nvda\include\comtypes\comtypes\client\_generate.py", line 110, in GetModule
    mod = _CreateWrapper(tlib, pathname)
  File "D:\my_repos\test\nvda\include\comtypes\comtypes\client\_generate.py", line 184, in _CreateWrapper
    mod = _my_import(fullname)
  File "D:\my_repos\test\nvda\include\comtypes\comtypes\client\_generate.py", line 24, in _my_import
    return __import__(fullname, globals(), locals(), ['DUMMY'])
ModuleNotFoundError: No module named 'comtypes.gen._8CC497C9_A1DF_11CE_8098_00AA0047BE5D_0_1_0'
scons: building terminated because of errors.

The third SCons run finishes successfully, however if creating launcher afterwards some dlls are missing - for example BRLApi.

### Expected behavior:
SCons should always work reliably.
### System configuration
#### NVDA installed/portable/running from source:
Source
#### NVDA version:
Current master as of 16th September of 2019.
#### Windows version:
Windows 7 x64
#### Name and version of other software in use when reproducing the issue:
Python  3.7.4, Visual Studio Community 2017
#### Other information about your system:
Nothing unusuall
### Other questions
#### Does the issue still occur after restarting your PC?
Yes
#### Have you tried any other versions of NVDA? If so, please report their behaviors.
I haven't experienced this with Python 2 based master.

This bug manifests also sometimes during AppVeyor builds, however it is random there.



Discussion:
> ### Steps to reproduce:
> 1. Clone fresh NVDA repo passing --recursive option as described in the readme.
> 2. Execute scons source from the root of the repo.
> 
> ### Actual behavior:
> Output of first scons run ends with the following:
> scons: *** [source\comInterfaces\SpeechLib.py] ModuleNotFoundError : No module named 'comtypes.gen._C866CA3A_32F7_11D2_9602_00C04F8EE628_0_5_4'
> Traceback (most recent call last):
> File "D:\my_repos\test\nvda\include\scons\src\engine\SCons\Action.py", line 1202, in execute
> result = self.execfunction(target=target, source=rsources, env=env)
> File "D:\my_repos\test\nvda\source\comInterfaces_sconscript", line 27, in interfaceAction
> comtypes.client.GetModule(source)
> File "D:\my_repos\test\nvda\include\comtypes\comtypes\client_generate.py", line 110, in GetModule
> mod = _CreateWrapper(tlib, pathname)
> File "D:\my_repos\test\nvda\include\comtypes\comtypes\client_generate.py", line 184, in _CreateWrapper
> mod = _my_import(fullname)
> File "D:\my_repos\test\nvda\include\comtypes\comtypes\client_generate.py", line 24, in _my_import
> return **import**(fullname, globals(), locals(), ['DUMMY'])
> ModuleNotFoundError: No module named 'comtypes.gen._C866CA3A_32F7_11D2_9602_00C04F8EE628_0_5_4'
> scons: building terminated because of errors.
> 
> Output of second scons run ends with the following:
> 
> scons: *** [source\comInterfaces\tom.py] ModuleNotFoundError : No module named 'comtypes.gen._8CC497C9_A1DF_11CE_8098_00AA0047BE5D_0_1_0'
> Traceback (most recent call last):
> File "D:\my_repos\test\nvda\include\scons\src\engine\SCons\Action.py", line 1202, in execute
> result = self.execfunction(target=target, source=rsources, env=env)
> File "D:\my_repos\test\nvda\source\comInterfaces_sconscript", line 27, in interfaceAction
> comtypes.client.GetModule(source)
> File "D:\my_repos\test\nvda\include\comtypes\comtypes\client_generate.py", line 110, in GetModule
> mod = _CreateWrapper(tlib, pathname)
> File "D:\my_repos\test\nvda\include\comtypes\comtypes\client_generate.py", line 184, in _CreateWrapper
> mod = _my_import(fullname)
> File "D:\my_repos\test\nvda\include\comtypes\comtypes\client_generate.py", line 24, in _my_import
> return **import**(fullname, globals(), locals(), ['DUMMY'])
> ModuleNotFoundError: No module named 'comtypes.gen._8CC497C9_A1DF_11CE_8098_00AA0047BE5D_0_1_0'
> scons: building terminated because of errors.
> 
> The third SCons run finishes successfully, however if creating launcher afterwards some dlls are missing - for example BRLApi.
> 
> ### Expected behavior:
> SCons should always work reliably.
> 
> ### System configuration
> #### NVDA installed/portable/running from source:
> Source
> 
> #### NVDA version:
> Current master as of 16th September of 2019.
> 
> #### Windows version:
> Windows 7 x64
> 
> #### Name and version of other software in use when reproducing the issue:
> Python 3.7.4, Visual Studio Community 2017
> 
> #### Other information about your system:
> Nothing unusuall
> 
> ### Other questions
> #### Does the issue still occur after restarting your PC?
> Yes
> 
> #### Have you tried any other versions of NVDA? If so, please report their behaviors.
> I haven't experienced this with Python 2 based master.
> 
> This bug manifests also sometimes during AppVeyor builds, however it is random there.

hi lukaszgo
You can delete the submodules of the include directory, and then update the submodules with git submodule update. I can compile windows10-1903 and visual studio2019 here.
> This bug manifests also sometimes during AppVeyor builds, however it is random there.

Yes, I've seen it as well when I started working on #10169. It is very hard to debug this on appveyor.

@lukaszgo1: could you verify whether.

1. The `_C866CA3A_32F7_11D2_9602_00C04F8EE628_0_5_4` module exists after you're getting the error about speechlib Note that the module should not only exist, the file should also be populated
2. Same for `_8CC497C9_A1DF_11CE_8098_00AA0047BE5D_0_1_0` for the tom module.

This is probably a bug in comtypes, I've seen this on running copies as well. My theory is that comtypes writes the module and then closes the file, but the file is not yet properly written to the file system before it tries to import it.

Note that #10169 mentions a workaround by ignoring ModuleNotFoundError, but that workaround is not yet in that code. If the files according to point 1 and 2 above exist though, I think it is safe to ignore these errors. We still need to point the comtypes authors at this problem, though.
@leonardder wrote:

>     1. The `_C866CA3A_32F7_11D2_9602_00C04F8EE628_0_5_4` module exists after you're getting the error about speechlib Note that the module should not only exist, the file should also be populated

the file _C866CA3A_32F7_11D2_9602_00C04F8EE628_0_5_4.py exists in source/comInterfaces, and is not empty if that is what you mean by ''populated'

> 
>     2. Same for `_8CC497C9_A1DF_11CE_8098_00AA0047BE5D_0_1_0` for the tom module.


This one also exists and is not empty.


Yes, indeed. So comtypes writes the module just fine.

It looks like there is a fix for this in https://github.com/enthought/comtypes/pull/172

Resolving patch:
diff --git a/source/comtypesMonkeyPatches.py b/source/comtypesMonkeyPatches.py
--- a/source/comtypesMonkeyPatches.py
+++ b/source/comtypesMonkeyPatches.py
@@ -1,10 +1,12 @@
-#A part of NonVisual Desktop Access (NVDA)
-#Copyright (C) 2009-2016 NV Access Limited
-#This file is covered by the GNU General Public License.
-#See the file COPYING for more details.
+# A part of NonVisual Desktop Access (NVDA)
+# Copyright (C) 2009-2019 NV Access Limited, Babbage B.V.
+# This file is covered by the GNU General Public License.
+# See the file COPYING for more details.
 
 import ctypes
 import _ctypes
+import importlib
+import comtypes.client._generate
 
 # A version of ctypes.WINFUNCTYPE 
 # that produces a WinFunctionType class whose instance will convert COMError into a CallCancelled exception when called as a function.
@@ -125,3 +127,15 @@ def _check_version(actual):
 	if actual != required:
 		raise ImportError("Wrong version")
 comtypes._check_version = _check_version
+
+
+# Monkeypatch comtypes to clear the importlib cache when importing a new module
+old_my_import = comtypes.client._generate._my_import
+
+
+def new_my_import(fullname):
+	importlib.invalidate_caches()
+	return old_my_import(fullname)
+
+
+comtypes.client._generate._my_import = new_my_import


Test patch: