Repository: numba/numba

Issue:
can't create pycc AOT compiler on Windows
<!--

Thanks for opening an issue! To help the Numba team handle your information
efficiently, please first ensure that there is no other issue present that
already describes the issue you have
(search at https://github.com/numba/numba/issues?&q=is%3Aissue).

For more general "how do I do X?" type questions, please speak to us in real
time on https://gitter.im/numba/numba or post to the Numba mailing list
https://groups.google.com/a/continuum.io/forum/#!forum/numba-users.

-->

## Reporting a bug

<!--

Before submitting a bug report please ensure that you can check off these boxes:

-->

- [x] I am using the latest released version of Numba (most recent is visible in
 the change log (https://github.com/numba/numba/blob/master/CHANGE_LOG).
- [x] I have included below a minimal working reproducer (if you are unsure how
 to write one see http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports).

<!--

Please include details of the bug here, including, if applicable, what you
expected to happen!

-->
numba.pycc.CC fails upon initialization on windows, despite having a compiler (distutils works fine):

    In  [1]: from numba.pycc import CC
    In  [2]: cc = CC('foo')
    Traceback (most recent call last):
      File "<ipython-input-2-06608376f747>", line 1, in <module>
        cc = CC('foo')
      File "C:\Anaconda3\lib\site-packages\numba\pycc\cc.py", line 62, in __init__
        self._toolchain = Toolchain()
      File "C:\Anaconda3\lib\site-packages\numba\pycc\platform. line 63, in __init__
        self._raise_external_compiler_error()
      File "C:\Anaconda3\lib\site-packages\numba\pycc\platform. line 106, in _raise_external_compiler_error
        raise RuntimeError(msg)
    RuntimeError: Attempted to compile AOT function without the compiler used
    by `numpy.distutils` present. Cannot find suitable msvc.

I tracked the cause down to numba.pycc.platform._check_external_compiler, lines 37-46.
On windows `tempfile` locks its file so cl.exe can't access it (see https://stackoverflow.com/q/20328422/9151228 & https://stackoverflow.com/q/14388608/9151228).
Numba mistakes this for the lack of a compiler.
Using TemporaryDirectory instead of NamedTemporaryFile seems to work for me:

    from distutils.sysconfig import customize_compiler
    from tempfile import NamedTemporaryFile, gettempdir, TemporaryDirectory
    import os.path as osp
    
    def check_compiler_file():
        # see if the external compiler bound in numpy.distutil is present
        # and working
        compiler = new_compiler()
        customize_compiler(compiler)
        for suffix in ['.c', '.cxx']:
            with NamedTemporaryFile('wt', suffix=suffix) as ntf:
                simple_c = "int main(void) { return 0; }"
                ntf.write(simple_c)
                ntf.flush()
                try:
                    # *output_dir* is set to avoid the compiler putting temp files
                    # in the current directory.
                    compiler.compile([ntf.name], output_dir=gettempdir())
                except Exception as exc:  # likely CompileError
                    return exc
        return None
    
    def check_compiler_dir():
        # see if the external compiler bound in numpy.distutil is present
        # and working
        compiler = new_compiler()
        customize_compiler(compiler)
        for suffix in ['.c', '.cxx']:
            with TemporaryDirectory() as tdr:
                fname = osp.join(tdr, "temp" + suffix)
                simple_c = "int main(void) { return 0; }"
                f = open(fname, 'wt')
                f.write(simple_c)
                f.flush()
                f.close()
                try:
                    # *output_dir* is set to avoid the compiler putting temp files
                    # in the current directory.
                    compiler.compile([fname], output_dir=tdr)
                except Exception as exc:  # likely CompileError
                    return exc
        return None

On windows, when there is a compiler available:

    In  [1]: check_compiler_file()
    Out [1]: CompileError(DistutilsExecError("command 'C:\\...\\cl.exe' failed
    with exit status 2"))
    In  [2]: check_compiler_dir()
    Out [2]:

When there is no compiler::

    In  [1]: check_compiler_file()
    Out [1]: DistutilsPlatformError('Unable to find vcvarsall.bat')
    In  [2]: check_compiler_dir()
    Out [2]: DistutilsPlatformError('Unable to find vcvarsall.bat')

Environment:

    import sys
    import numba
    import platform
    print(sys.version)
    print(numba.__version__)
    print(platform.platform())
    3.7.0 (default, Jun 28 2018, 08:04:48) [MSC v.1912 64 bit (AMD64)]
    0.39.0
    Windows-10-10.0.17134-SP0

Numba 0.40.0 hasn't made it to anaconda yet, but the relevant part of platform.py looks the same to me.
I managed to hack my way around this (and didn't wind up using the AOT compiler) so this issue doesn't really affect me, but it seemed worth reporting anyway.


Discussion:
Thanks for the bug report, and the investigation.  We'll need to see why this isn't caught by our unit tests...

Resolving patch:
diff --git a/numba/pycc/platform.py b/numba/pycc/platform.py
--- a/numba/pycc/platform.py
+++ b/numba/pycc/platform.py
@@ -11,7 +11,8 @@
 import os
 import subprocess
 import sys
-from tempfile import NamedTemporaryFile, gettempdir
+from tempfile import NamedTemporaryFile, mkdtemp, gettempdir
+from contextlib import contextmanager
 
 _configs = {
     # DLL suffix, Python C extension suffix
@@ -27,6 +28,22 @@ def get_configs(arg):
 find_shared_ending = functools.partial(get_configs, 0)
 find_pyext_ending = functools.partial(get_configs, 1)
 
+@contextmanager
+def _gentmpfile(suffix):
+    # windows locks the tempfile so use a tempdir + file, see
+    # https://github.com/numba/numba/issues/3304
+    try:
+        tmpdir = mkdtemp()
+        ntf = open(os.path.join(tmpdir, "temp%s" % suffix), 'wt')
+        yield ntf
+    finally:
+        try:
+            ntf.close()
+            os.remove(ntf)
+        except:
+            pass
+        else:
+            os.rmdir(tmpdir)
 
 def _check_external_compiler():
     # see if the external compiler bound in numpy.distutil is present
@@ -34,19 +51,19 @@ def _check_external_compiler():
     compiler = new_compiler()
     customize_compiler(compiler)
     for suffix in ['.c', '.cxx']:
-        with NamedTemporaryFile('wt', suffix=suffix) as ntf:
-            simple_c = "int main(void) { return 0; }"
-            ntf.write(simple_c)
-            ntf.flush()
-            try:
+        try:
+            with _gentmpfile(suffix) as ntf:
+                simple_c = "int main(void) { return 0; }"
+                ntf.write(simple_c)
+                ntf.flush()
+                ntf.close()
                 # *output_dir* is set to avoid the compiler putting temp files
                 # in the current directory.
                 compiler.compile([ntf.name], output_dir=gettempdir())
-            except Exception: # likely CompileError
-                return False
+        except Exception: # likely CompileError or file system issue
+            return False
     return True
 
-
 # boolean on whether the externally provided compiler is present and
 # functioning correctly
 _external_compiler_ok = _check_external_compiler()


Test patch:
diff --git a/numba/tests/test_pycc.py b/numba/tests/test_pycc.py
--- a/numba/tests/test_pycc.py
+++ b/numba/tests/test_pycc.py
@@ -28,6 +28,9 @@
 _skip_reason = 'AOT compatible compilers missing'
 _skip_missing_compilers = unittest.skipIf(not _external_compiler_ok,
                                           _skip_reason)
+_skip_reason = 'windows only'
+_windows_only = unittest.skipIf(not sys.platform.startswith('win'),
+                                _skip_reason)
 
 from .matmul_usecase import has_blas
 from .support import TestCase, tag, import_dynamic, temp_directory
@@ -43,6 +46,19 @@ def unset_macosx_deployment_target():
     if 'MACOSX_DEPLOYMENT_TARGET' in os.environ:
         del os.environ['MACOSX_DEPLOYMENT_TARGET']
 
+class TestCompilerChecks(TestCase):
+
+    # NOTE: THIS TEST MUST ALWAYS RUN ON WINDOWS, DO NOT SKIP
+    @_windows_only
+    def test_windows_compiler_validity(self):
+        # When inside conda-build VSINSTALLDIR should be set and windows should
+        # have a valid compiler available, `_external_compiler_ok` should  agree
+        # with this. If this is not the case then error out to alert devs.
+        is_running_conda_build = os.environ.get('CONDA_BUILD', None) is not None
+        if is_running_conda_build:
+            if os.environ.get('VSINSTALLDIR', None) is not None:
+                self.assertTrue(_external_compiler_ok)
+
 
 class BasePYCCTest(TestCase):