Repository: cupy/cupy
Pull request: Import numpy before Cython
Closes #5476 

Thanks to @kmaehashi for the alternative solution

Problem statement:
Cupy cannot be compiled depending on the combination of versions of Setuptools, Numpy, Cython
It seems that Cupy cannot be compiled depending on the combination of versions of Setuptools, Numpy, Cython (and other components).

* Conditions 
  - CuPy 9.2.0 (try to compile)
  - Linux OS 5.13.1 / glibc 2.33
  - CUDA 11.4 / NVIDIA driver 470.42.01 (Beta driver which supports CUDA 11.4)
  - Python 3.9.6 / Cython 0.29.33
  - Setuptools 57.1.0 / Numpy 1.21.0 
  - GCC 10.2.0 (..also tried using  GCC9)
* Error messages  that came up when I tried running "DISTUTILS_DEBUG=1 python setup.py build".

```
[61/63] Cythonizing cupy_backends/cuda/libs/nvtx.pyx
[62/63] Cythonizing cupy_backends/cuda/libs/profiler.pyx
[63/63] Cythonizing cupy_backends/cuda/stream.pyx
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/numpy/distutils/ccompiler.py", line 735, in new_compiler
    (module_name, class_name, long_description) = compiler_class[compiler]
KeyError: 'nvidia'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/XXX/python-cupy/src/cupy-9.2.0/setup.py", line 145, in <module>
    setup(
  File "/usr/lib/python3.9/site-packages/setuptools/__init__.py", line 153, in setup
    return distutils.core.setup(**attrs)
  File "/usr/lib/python3.9/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/lib/python3.9/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python3.9/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/usr/lib/python3.9/distutils/command/build.py", line 135, in run
    self.run_command(cmd_name)
  File "/usr/lib/python3.9/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/usr/lib/python3.9/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/XXX/python-cupy/src/cupy-9.2.0/cupy_setup_build.py", line 1222, in run
    build_ext.build_ext.run(self)
  File "/usr/lib/python3.9/site-packages/setuptools/command/build_ext.py", line 79, in run
    _build_ext.run(self)
  File "/usr/lib/python3.9/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run
    _build_ext.build_ext.run(self)
  File "/usr/lib/python3.9/distutils/command/build_ext.py", line 306, in run
    self.compiler = new_compiler(compiler=self.compiler,
  File "/usr/lib/python3.9/site-packages/numpy/distutils/ccompiler.py", line 740, in new_compiler
    raise DistutilsPlatformError(msg)
distutils.errors.DistutilsPlatformError: don't know how to compile C/C++ code on platform 'posix' with 'nvidia' compiler

```

Hints from issue discussion:
Thanks for reporting! According to the logs it seems `import numpy` is happening during CuPy build. NumPy [monkypatches](https://github.com/numpy/numpy/blob/df9c973f28fb9c0e713db29285b07b57c622652a/numpy/distutils/ccompiler.py#L766) the distutils so if numpy is imported during the CuPy build it can fail.
I think you can try `echo 'raise RuntimeError' > numpy.py && python setup.py build` to see who actually does `import numpy` 😄 
@kmaehashi -san

Here is the message I retrieved in  trying to build cupy on your guidance...


```
************************************************************
* CuPy Configuration Summary                               *
************************************************************

Build Environment:
  Include directories: ['/opt/cuda/include/cub', '/XXX/python-cupy/src/cupy-9.2.0/install/../cupy/_core/include', '/opt/cuda/include']
  Library directories: ['/opt/cuda/lib64']
  nvcc command       : ['/opt/cuda/bin/nvcc']
  hipcc command      : (not found)

Environment Variables:
  CFLAGS          : -march=skylake-avx512 -mtune=native -O3 -pipe -fno-plt -fexceptions         -Wp,-D_FORTIFY_SOURCE=2,-D_GLIBCXX_ASSERTIONS         -Wformat -Werror=format-security         -fstack-clash-protection -fcf-protection -I/usr/include/cugraph 
  LDFLAGS         : -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now
  LIBRARY_PATH    : (none)
  CUDA_PATH       : /opt/cuda
  NVTOOLSEXT_PATH : (none)
  NVCC            : (none)
  HIPCC           : (none)
  ROCM_HOME       : (none)

Modules:
  cuda      : Yes (version 11040)
  cusolver  : Yes
  cudnn     : Yes (version 8201)
  nccl      : Yes (version 20909)
  nvtx      : Yes
  cutensor  : Yes (version 1300)
  cub       : Yes (version 101201)
  jitify    : Yes (version <unknown>)
  random    : Yes
  cusparselt: Yes (version 100)
  cugraph   : No
    -> Include files not found: ['cugraph/raft/error.hpp']
    -> Check your CFLAGS environment variable.
  thrust    : Yes (version 101201)
  dlpack    : Yes

WARNING: Some modules could not be configured.
CuPy will be installed without these modules.
Please refer to the Installation Guide for details:
https://docs.cupy.dev/en/stable/install.html

************************************************************

Traceback (most recent call last):
  File "/XXX/python-cupy/src/cupy-9.2.0/setup.py", line 145, in <module>
    setup(
  File "/usr/lib/python3.9/site-packages/setuptools/__init__.py", line 153, in setup
    return distutils.core.setup(**attrs)
  File "/usr/lib/python3.9/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/lib/python3.9/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python3.9/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/usr/lib/python3.9/distutils/command/build.py", line 135, in run
    self.run_command(cmd_name)
  File "/usr/lib/python3.9/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/usr/lib/python3.9/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/XXX/python-cupy/src/cupy-9.2.0/cupy_setup_build.py", line 1220, in run
    cythonize(ext_modules, cupy_setup_options)
  File "/XXX/python-cupy/src/cupy-9.2.0/cupy_setup_build.py", line 929, in cythonize
    return Cython.Build.cythonize(
  File "/usr/lib/python3.9/site-packages/Cython/Build/Dependencies.py", line 963, in cythonize
    ctx = c_options.create_context()
  File "/usr/lib/python3.9/site-packages/Cython/Compiler/Main.py", line 596, in create_context
    return Context(self.include_path, self.compiler_directives,
  File "/usr/lib/python3.9/site-packages/Cython/Compiler/Main.py", line 80, in __init__
    from . import Builtin, CythonScope
  File "/usr/lib/python3.9/site-packages/Cython/Compiler/CythonScope.py", line 5, in <module>
    from .UtilityCode import CythonUtilityCode
  File "/usr/lib/python3.9/site-packages/Cython/Compiler/UtilityCode.py", line 3, in <module>
    from .TreeFragment import parse_from_strings, StringParseContext
  File "/usr/lib/python3.9/site-packages/Cython/Compiler/TreeFragment.py", line 17, in <module>
    from .Visitor import VisitorTransform
  File "Cython/Compiler/Visitor.py", line 16, in init Cython.Compiler.Visitor
  File "/usr/lib/python3.9/site-packages/Cython/Compiler/Nodes.py", line 31, in <module>
    from .Pythran import has_np_pythran, pythran_type, is_pythran_buffer
  File "/usr/lib/python3.9/site-packages/Cython/Compiler/Pythran.py", line 10, in <module>
    import pythran
  File "/usr/lib/python3.9/site-packages/pythran/__init__.py", line 40, in <module>
    from pythran.toolchain import (generate_cxx, compile_cxxfile, compile_cxxcode,
  File "/usr/lib/python3.9/site-packages/pythran/toolchain.py", line 6, in <module>
    from pythran.backend import Cxx, Python
  File "/usr/lib/python3.9/site-packages/pythran/backend.py", line 7, in <module>
    from pythran.analyses import LocalNodeDeclarations, GlobalDeclarations, Scope
  File "/usr/lib/python3.9/site-packages/pythran/analyses/__init__.py", line 12, in <module>
    from .aliases import Aliases, StrictAliases
  File "/usr/lib/python3.9/site-packages/pythran/analyses/aliases.py", line 4, in <module>
    from pythran.intrinsic import Intrinsic, Class, UnboundValue
  File "/usr/lib/python3.9/site-packages/pythran/intrinsic.py", line 3, in <module>
    from pythran.conversion import to_ast
  File "/usr/lib/python3.9/site-packages/pythran/conversion.py", line 4, in <module>
    import numpy as np
  File "/XXX/python-cupy/src/cupy-9.2.0/numpy.py", line 1, in <module>
    raise RuntimeError
RuntimeError

```

I reproduced this issue,

just by having `pythran` installed, cupy can't be built

Commit message: import distools instead

Patch to review:
diff --git a/cupy_setup_build.py b/cupy_setup_build.py
index d1401f5ad04..483bd5d73b7 100644
--- a/cupy_setup_build.py
+++ b/cupy_setup_build.py
@@ -18,6 +18,12 @@
 from install.build import PLATFORM_LINUX
 from install.build import PLATFORM_WIN32
 
+try:
+    # This is to avoid getting numpy imported inside other modules and
+    # overwritting setuptools compilers
+    import numpy.distutils  # NOQA
+except Exception:
+    pass
 
 # Cython requirements (minimum version and versions known to be broken).
 # Note: this must be in sync with setup_requires defined in setup.py.


Code review comments:
diff_hunk: @@ -18,6 +18,12 @@
 from install.build import PLATFORM_LINUX
 from install.build import PLATFORM_WIN32
 
+try:
+    # This is to avoid getting numpy imported inside other modules and
+    # overwritting setuptools compilers
path: cupy_setup_build.py
text: @user1:
```suggestion
    # This is to avoid getting numpy imported inside other modules and
    # overwritting setuptools compilers (#5476)
```