Repository: numba/numba

Issue:
Compiler not found: /tmp/tmp bug


```
$ pip install numba
Collecting numba
  Downloading numba-0.53.1-cp39-cp39-manylinux2014_x86_64.whl (3.4 MB)
     |████████████████████████████████| 3.4 MB 1.2 MB/s 
Collecting numpy>=1.15
  Downloading numpy-1.21.0-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.7 MB)
     |████████████████████████████████| 15.7 MB 1.1 MB/s 
Requirement already satisfied: setuptools in /usr/lib/python3/dist-packages (from numba) (52.0.0)
Collecting llvmlite<0.37,>=0.36.0rc1
  Downloading llvmlite-0.36.0-cp39-cp39-manylinux2010_x86_64.whl (25.3 MB)
     |████████████████████████████████| 25.3 MB 829 kB/s 
Installing collected packages: numpy, llvmlite, numba
  WARNING: The scripts f2py, f2py3 and f2py3.9 are installed in '/home/olaf/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed llvmlite-0.36.0 numba-0.53.1 numpy-1.21.0

$ ./t.py 
          
$ ./t.py 

$ chmod 000 /tmp/tmp

$ ./t.py            
Traceback (most recent call last):
  File "/home/olaf/./t.py", line 5, in <module>
    cc = CC('interpolation')
  File "/home/olaf/.local/lib/python3.9/site-packages/numba/pycc/cc.py", line 65, in __init__
    self._toolchain = Toolchain()
  File "/home/olaf/.local/lib/python3.9/site-packages/numba/pycc/platform.py", line 78, in __init__
    self._raise_external_compiler_error()
  File "/home/olaf/.local/lib/python3.9/site-packages/numba/pycc/platform.py", line 121, in _raise_external_compiler_error
    raise RuntimeError(msg)
RuntimeError: Attempted to compile AOT function without the compiler used by `numpy.distutils` present. If using conda try:

#> conda install gcc_linux-64 gxx_linux-64

$ cat t.py 
#!/usr/bin/python3

from numba.pycc import CC

cc = CC('interpolation')

# cc.compile()

$ pip list|grep numba
numba            0.53.1
```


Discussion:
You'll need a compiler in your system to use the AOT support.
I know, it's there, this is a bug report, trust me. ;)
The following use of distutils exhibits the same behaviour when `/tmp/tmp` has `000` permissions:

```python
from distutils.ccompiler import new_compiler
from distutils.sysconfig import customize_compiler
from tempfile import mkdtemp, gettempdir
import os

compiler = new_compiler()
customize_compiler(compiler)

tmpdir = mkdtemp()

with open(os.path.join(tmpdir, "temp.c"), 'wt') as f:
    simple_c = "int main(void) { return 0; }"
    f.write(simple_c)
    f.flush()
    f.close()
    compiler.compile([f.name], output_dir=gettempdir())
```

@OlafvdSpek Do you think there is something wrong with the above example? If so, what do you think Numba should do when `/tmp/tmp` is `000`?
>     compiler.compile([f.name], output_dir=gettempdir())

Hi Graham,

I think the default output_dir is fine / better, so no need to pass output_dir. If you do want to set the output_dir, IMO you should use output_dir=tmpdir.
I'm also wondering whether this compiler check could be skipped altogether.
Also, it should clean up the temp files / dirs it created.
> I think the default output_dir is fine / better, so no need to pass output_dir. If you do want to set the output_dir, IMO you should use output_dir=tmpdir.

We don't use the default dir because of https://github.com/numba/numba/issues/3304.

> I'm also wondering whether this compiler check could be skipped altogether.

I don't think this would be a good idea because the compiler may not be present, and we need to try and give a reasonable error message. Perhaps the error message could be improved, but I think we need to keep the check.

The example I have above gives:

```
Traceback (most recent call last):
  File "/home/gmarkall/miniconda3/envs/numbanp120/lib/python3.9/distutils/dir_util.py", line 70, in mkpath
    os.mkdir(head, mode)
PermissionError: [Errno 13] Permission denied: '/tmp/tmp/tmpc3ueagd8'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/gmarkall/numbadev/issues/7218/du.py", line 16, in <module>
    compiler.compile([f.name], output_dir=gettempdir())
  File "/home/gmarkall/miniconda3/envs/numbanp120/lib/python3.9/distutils/ccompiler.py", line 565, in compile
    self._setup_compile(output_dir, macros, include_dirs, sources,
  File "/home/gmarkall/miniconda3/envs/numbanp120/lib/python3.9/distutils/ccompiler.py", line 348, in _setup_compile
    self.mkpath(os.path.dirname(obj))
  File "/home/gmarkall/miniconda3/envs/numbanp120/lib/python3.9/distutils/ccompiler.py", line 916, in mkpath
    mkpath(name, mode, dry_run=self.dry_run)
  File "/home/gmarkall/miniconda3/envs/numbanp120/lib/python3.9/distutils/dir_util.py", line 73, in mkpath
    raise DistutilsFileError(
distutils.errors.DistutilsFileError: could not create '/tmp/tmp/tmpc3ueagd8': Permission denied
```

So in this case we should try to pass this error message back to the user, rather than stating that the compiler is not present.

> Also, it should clean up the temp files / dirs it created.

You're right here - I distilled the above example from the code in Numba, which does clean up the files and dir it makes - I ended up omitting that for the example
> > I think the default output_dir is fine / better, so no need to pass output_dir. If you do want to set the output_dir, IMO you should use output_dir=tmpdir.
> 
> We don't use the default dir because of #3304.

OK, what about "If you do want to set the output_dir, IMO you should use output_dir=tmpdir." ?

> > I'm also wondering whether this compiler check could be skipped altogether.
> 
> I don't think this would be a good idea because the compiler may not be present, and we need to try and give a reasonable error message. Perhaps the error message could be improved, but I think we need to keep the check.

Can't you return that error when an actual compile step fails?
 
> So in this case we should try to pass this error message back to the user, rather than stating that the compiler is not present.

True
 
> > Also, it should clean up the temp files / dirs it created.
> 
> You're right here - I distilled the above example from the code in Numba, which does clean up the files and dir it makes - I ended up omitting that for the example

The clean up doesn't appear to be working for me, I'm left with:

```
/tmp/tmp/tmpructlr3z/temp.o
/tmp/tmp/tmpuic9pxgb/temp.o
/tmp/tmpructlr3z/temp.cxx
/tmp/tmpuic9pxgb/temp.c
```
Hi,

We have just been seeing this issue on our end, we are compiling numba code AOT on a HPC, so whoever compiles numba first will create a /tmp/tmp directory with drwxrwxr-x permissions which means that attempts to compile numba will fail for anyone else. This is will be problematic for us as we will want to distribute our code to other users and use it on other HPC resources so that we will be seeing it quite often.

Will there be a solution for this issue?
@ChiCheng45 `chmod 777 /tmp/tmp`
Hi @OlafvdSpek yes that would fix the issue but we distribute our code to other people on different HPCs etc which will mean that this issue will be flagged to us multiple times. I have proposed a fix.

Resolving patch:
diff --git a/numba/pycc/platform.py b/numba/pycc/platform.py
--- a/numba/pycc/platform.py
+++ b/numba/pycc/platform.py
@@ -9,8 +9,9 @@
 import os
 import subprocess
 import sys
-from tempfile import NamedTemporaryFile, mkdtemp, gettempdir
+from tempfile import mkdtemp
 from contextlib import contextmanager
+from pathlib import Path
 
 _configs = {
     # DLL suffix, Python C extension suffix
@@ -57,7 +58,7 @@ def _check_external_compiler():
                 ntf.close()
                 # *output_dir* is set to avoid the compiler putting temp files
                 # in the current directory.
-                compiler.compile([ntf.name], output_dir=gettempdir())
+                compiler.compile([ntf.name], output_dir=Path(ntf.name).anchor)
         except Exception: # likely CompileError or file system issue
             return False
     return True


Test patch: