Repository: cupy/cupy

Issue:
cupy 2.2.0 with cudnn support: installation fails with cython 0.25.2
When installing cupy 2.2.0 when cudnn is present, installation fails depending on the Cython version that is present.

To reproduce, check the following Dockerfile:

```
FROM nvidia/cuda:8.0-cudnn7-devel-ubuntu16.04
RUN apt-get update && apt-get install -y python3-pip python3-dev
RUN pip3 install pip==9.0.1
RUN pip3 install requests[security]==2.13.0 setuptools==34.1.0
RUN pip3 install Cython==0.25.2
RUN pip3 install cupy==2.2.0
```

This fails with the following error:

```
    [1/3] Cythonizing cupy/cudnn.pyx
    Compiling /tmp/pip-build-z0vd49d6/cupy/cupy/cudnn.pyx
    
    Error compiling Cython file:
    ------------------------------------------------------------
    ...
            compute_type = cudnn.CUDNN_DATA_FLOAT
    
        if ndim != 2:
            c_pad = pad
            c_stride = stride
            c_dilation.assign(ndim, 1)
                            ^
    ------------------------------------------------------------
    
    cupy/cudnn.pyx:133:25: Compiler crash in AnalyseExpressionsTransform
    
    ModuleNode.body = StatListNode(cudnn.pyx:1:0)
    StatListNode.stats[21] = StatListNode(cudnn.pyx:115:6)
    StatListNode.stats[0] = CFuncDefNode(cudnn.pyx:115:6,
        args = [...]/8,
        modifiers = [...]/0,
        overridable = 1,
        visibility = 'private')
    File 'Nodes.py', line 430, in analyse_expressions: StatListNode(cudnn.pyx:118:4)
    File 'Nodes.py', line 5845, in analyse_expressions: IfStatNode(cudnn.pyx:130:4)
    File 'Nodes.py', line 5891, in analyse_expressions: IfClauseNode(cudnn.pyx:130:7,
        is_terminator = True)
    File 'Nodes.py', line 430, in analyse_expressions: StatListNode(cudnn.pyx:131:8,
        is_terminator = True)
    File 'Nodes.py', line 4746, in analyse_expressions: ExprStatNode(cudnn.pyx:133:25)
    File 'ExprNodes.py', line 519, in analyse_expressions: SimpleCallNode(cudnn.pyx:133:25,
        analysed = True,
        use_managed_ref = True)
    File 'ExprNodes.py', line 5132, in analyse_types: SimpleCallNode(cudnn.pyx:133:25,
        analysed = True,
        use_managed_ref = True)
    File 'ExprNodes.py', line 5187, in analyse_c_function_call: SimpleCallNode(cudnn.pyx:133:25,
        analysed = True,
        use_managed_ref = True)
    
    Compiler crash traceback from this point on:
      File "/usr/local/lib/python3.5/dist-packages/Cython/Compiler/ExprNodes.py", line 5187, in analyse_c_function_call
        [arg.type for arg in args], alternatives, self.pos, env, args)
      File "/usr/local/lib/python3.5/dist-packages/Cython/Compiler/PyrexTypes.py", line 4083, in best_match
        errors.append((func, "Unable to deduce type parameters for %s given %s" % (pattern.type, actual)))
    NameError: name 'pattern' is not defined
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-z0vd49d6/cupy/setup.py", line 102, in <module>
        'sdist': sdist},
      File "/usr/lib/python3.5/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/usr/lib/python3.5/distutils/dist.py", line 955, in run_commands
        self.run_command(cmd)
      File "/usr/lib/python3.5/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/usr/local/lib/python3.5/dist-packages/setuptools/command/install.py", line 61, in run
        return orig.install.run(self)
      File "/usr/lib/python3.5/distutils/command/install.py", line 583, in run
        self.run_command('build')
      File "/usr/lib/python3.5/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/usr/lib/python3.5/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/usr/lib/python3.5/distutils/command/build.py", line 135, in run
        self.run_command(cmd_name)
      File "/usr/lib/python3.5/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/usr/lib/python3.5/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/tmp/pip-build-z0vd49d6/cupy/cupy_setup_build.py", line 552, in run
        cythonize(ext_modules, cupy_setup_options)
      File "/tmp/pip-build-z0vd49d6/cupy/cupy_setup_build.py", line 354, in cythonize
        compiler_directives=directives, **cythonize_options)
      File "/usr/local/lib/python3.5/dist-packages/Cython/Build/Dependencies.py", line 934, in cythonize
        cythonize_one(*args)
      File "/usr/local/lib/python3.5/dist-packages/Cython/Build/Dependencies.py", line 1056, in cythonize_one
        raise CompileError(None, pyx_file)
    Cython.Compiler.Errors.CompileError: cupy/cudnn.pyx
    
    ----------------------------------------
Command "/usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-z0vd49d6/cupy/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-5b_xwzzp-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-z0vd49d6/cupy/
```

I would expect that it doesn't fail because the Cython version is > 0.24.0, as written in the cupy documentation at https://docs-cupy.chainer.org/en/stable/install.html#install-cupy-for-developers

Note the following:
- The error goes away if the cupy version is replaced by 2.1.0.
- The error persists if the cupy version is replaced by 2.3.0.
- The error goes away if the cupy version is replaced by 3.0.0a1.
- The error goes away if the cython version is replaced by 0.25.1. (!!!)
- The error goes away if the cython version is replaced by 0.26.0.
- The error goes away if cython is not present.
- The error goes away if cudnn is not present.

I don't know what is a good way to deal with that. Either releasing 2.2.1 and 2.3.1 with an explicit Cython 0.25.2 check, or just updating the documentation, maybe?

Related:
- Cython changelog for 0.25.2.: https://github.com/cython/cython/blob/master/CHANGES.rst#0252-2016-12-08
- Other user with similar issue (Chainer): https://github.com/chainer/chainer/issues/2250
- Other user with similar issue (Cython): https://github.com/cython/cython/issues/1583


Discussion:
Thanks for reporting the issue, I think we shouldn't re-cythonize `*.pyx` files when installed via pip.

> Users need not to install Cython as a distribution package of CuPy only contains generated sources.

[The document](https://docs-cupy.chainer.org/en/stable/install.html#install-cupy-for-developers) is also incorrect; the package includes `*.pyx` files along with generated sources files.

Resolving patch:
diff --git a/cupy_setup_build.py b/cupy_setup_build.py
--- a/cupy_setup_build.py
+++ b/cupy_setup_build.py
@@ -160,8 +160,15 @@ def module_extension_name(file):
 
 def module_extension_sources(file, use_cython, no_cuda):
     pyx, others = ensure_module_file(file)
-    ext = '.pyx' if use_cython else '.cpp'
-    pyx = path.join(*pyx.split('.')) + ext
+    base = path.join(*pyx.split('.'))
+    if use_cython:
+        pyx = base + '.pyx'
+        if not os.path.exists(pyx):
+            use_cython = False
+            print(
+                'NOTICE: Skipping cythonize as {} does not exist.'.format(pyx))
+    if not use_cython:
+        pyx = base + '.cpp'
 
     # If CUDA SDK is not available, remove CUDA C files from extension sources
     # and use stubs defined in header files.


Test patch: