Repository: cython/cython

Issue:
build_ext does not properly track dependencies (and should be deprecated / removed)
Hello up there. In the context of getting [gevent miscompiled](https://github.com/gevent/gevent/issues/1568#issuecomment-617432757) the question of proper dependency tracking on Cython side was [raised](https://github.com/gevent/gevent/issues/1568#issuecomment-617636428). While original cause for that particular gevent miscompilation seems to be https://github.com/cython/cython/issues/1428, other issues were also found.

This issue is probably related to https://github.com/cython/cython/issues/1436 and shows that `Cython.Distutils.build_ext` does not properly track build dependencies.

In 2016 `Cython.Distutils.build_ext` was changed to use `cythonize` and the old implementation was moved into `Cython.Distutils.old_build_ext` for exactly particular reason that `old_build_ext` was not tracking dependencies properly: cb55c11b60d. A warning corresponding to the move was added:
https://github.com/cython/cython/blob/3de7a4b8fb7ce045222e13ca02541f6a70e89c2e/Cython/Distutils/old_build_ext.py#L37-L42

However right after that `Cython.Distutils.build_ext` was reverted to use `old_build_ext` and new implementation became available as `new_build_ext` (4ecdd3e4) with the idea to

> give projects more time to move over, and reduces the number of changes in the 0.25 series

As of today (2020 April) `Cython.Distutils.build_ext` still points to `old_build_ext` and the warning is somehow **not** printed even if `old_build_ext` is imported directly(*):

```
$ python -c 'import Cython.Distutils.old_build_ext'
# empty output
$ python -c 'from Cython.Build import build_ext'
# empty output
```

This way many projects still use `old_build_ext` = `from Cython.Build import build_ext` to compile their pyx files and miss both proper dependency tracking and the warning.

As a fix I propose to make the deprecation warnings effective and to make `Cython.Build.build_ext` to refer to `new_build_ext` which uses cythonize.

The rest of this issue demonstrates that `Cython.Build.build_ext` dependency handling is broken.

--------

`setup.py`
```py
from setuptools import setup, Extension
from Cython.Distutils import build_ext

setup(
    ext_modules = [
                Extension("a", ["a.pyx"]),
                Extension("b", ["b.pyx"]),
            ],

    cmdclass = {'build_ext': build_ext}
)
```

`a.pyx`
```pyx
# cython: language_level=2
  
from b cimport bfunc

def afunc():
    bfunc()
```

`b.pxd`
```pyx
cdef bfunc()
```

`b.pyx`
```pyx
# cython: language_level=2

cdef bfunc():
    print "bbb"
```

( first compile - ok )
```
$ python setup.py build_ext -i
running build_ext
cythoning a.pyx to a.c
cythoning b.pyx to b.c
...
```

now change `b.pxd` and `b.pyx` to add argument to bfunc:

```diff
--- a/b.pxd
+++ b/b.pxd
@@ -1 +1 @@
-cdef bfunc()
+cdef bfunc(int x)
--- a/b.pyx
+++ b/b.pyx
@@ -1,4 +1,4 @@
 # cython: language_level=2
 
-cdef bfunc():
-    print "bbb"
+cdef bfunc(int x):
+    print "bbb", x
```

and recompile - only `b` is rebuilt - **not** `a`:

```
(neo) (z-dev) (g.env) kirr@deco:~/tmp/trashme/pyx$ python setup.py build_ext -i
running build_ext
skipping 'a.c' Cython extension (up-to-date)      <-- NOTE
cythoning b.pyx to b.c
/home/kirr/src/tools/py/cython/Cython/Compiler/Main.py:344: FutureWarning: Cython directive 'language_level' not set, using '3str' for now (Py3). This has changed from earlier releases! File: /home/kirr/tmp/trashme/pyx/b.pxd
  tree = Parsing.p_module(s, pxd, full_module_name)
building 'b' extension
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-2.7.16=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -c b.c -o build/temp.linux-x86_64-2.7/b.o
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-2.7.16=. -fstack-protector-strong -Wformat -Werror=format-security -Wl,-z,relro -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-2.7.16=. -fstack-protector-strong -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/b.o -o /home/kirr/tmp/trashme/pyx/b.so
```

which is **WRONG** because `a` will use `b` module via outdated pxd.

( In this particular case it should be giving compilation error at Cython level:

```
$ touch a.pyx 
$ python setup.py build_ext -i
running build_ext
cythoning a.pyx to a.c

Error compiling Cython file:
------------------------------------------------------------
...
# cython: language_level=2

from b cimport bfunc

def afunc():
    bfunc()
        ^
------------------------------------------------------------

a.pyx:6:9: Call with wrong number of arguments (expected 1, got 0)
```
)

Cython `3.0a1-79-g3de7a4b8f`

Thanks beforehand,
Kirill

/cc @robertwb 

P.S.

Cython documentation [says](https://cython.readthedocs.io/en/latest/src/userguide/source_files_and_compilation.html#distributing-cython-modules) to use `Cython.Build.new_build_ext` which should be using `cythonize` internally:

> Another option is to make Cython a setup dependency of your system and use Cython’s build_ext module which runs cythonize as part of the build process:
>
> ```py
> setup(
>     extensions = [Extension("*", ["*.pyx"])],
>     cmdclass={'build_ext': Cython.Build.new_build_ext},
>     ...
> )
> ```

however [Cython.Build](https://github.com/cython/cython/blob/3.0a1-79-g3de7a4b8f/Cython/Build/__init__.py) does not have it:

```
>>> import Cython.Build
>>> Cython.Build.new_build_ext
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'new_build_ext'
```

because `new_build_ext` is defined only in [Cython.Distutils.build_ext](https://github.com/cython/cython/blob/3.0a1-79-g3de7a4b8f/Cython/Distutils/build_ext.py#L14)


--------

(*) probably due to some logic error in _check_stack calls. if I patch Cython locally with

```diff
--- a/Cython/Distutils/old_build_ext.py
+++ b/Cython/Distutils/old_build_ext.py
@@ -34,9 +34,7 @@ def _check_stack(path):
     pass
   return False
 
-if (not _check_stack('setuptools/extensions.py')
-    and not _check_stack('pyximport/pyxbuild.py')
-    and not _check_stack('Cython/Distutils/build_ext.py')):
+if 1:
     warnings.warn(
         "Cython.Distutils.old_build_ext does not properly handle dependencies "
         "and is deprecated.")
```

the warning is printed:

```
$ python -c 'import Cython.Distutils.old_build_ext'
/home/kirr/src/tools/py/cython/Cython/Distutils/old_build_ext.py:39: UserWarning: Cython.Distutils.old_build_ext does not properly handle dependencies and is deprecated.
  "Cython.Distutils.old_build_ext does not properly handle dependencies "
$ python -c 'from Cython.Build import build_ext'
/home/kirr/src/tools/py/cython/Cython/Distutils/old_build_ext.py:39: UserWarning: Cython.Distutils.old_build_ext does not properly handle dependencies and is deprecated.
  "Cython.Distutils.old_build_ext does not properly handle dependencies "
```


Discussion:
Thanks for the report. The thing is, `new_build_ext` is not compatible with the setup that many users probably configured for `old_build_ext`, specifically any command options.

What might work, I think, is to copy the current configuration options from `old_build_ext` to `new_build_ext`, and pass them over to `cythonize()` only if they were provided. Basically, integrate a legacy layer into `new_build_ext` that allows existing projects to build as before. We can issue a warning when we find legacy usages, but we shouldn't break the build for it.

PR very welcome.
Here is how the options for the old_build_ext (on the left hand side) roughly map to cythonize parameters (right hand side) used by the new_build_ext:

cython_cplus                    : language 
cython_create_listing        : N/A?
cython_line_directives       : N/A?
cython_include_dirs          : include_path (maybe?)
cython_directives              : compiler_directives 
cython_c_in_temp             : N/A?
cython_gen_pxi                 : N/A?
cython_gdb                      : N/A?
no_c_in_traceback             : N/A?
cython_compile_time_env : N/A?

Given how incompatible the two build_exts are, is it worth making the new_build_ext backwards compatible with the old_build_ext, or should we just switch to the new_build_ext as a possibly breaking change for Cython 3.0? 


Resolving patch:
diff --git a/Cython/Distutils/build_ext.py b/Cython/Distutils/build_ext.py
--- a/Cython/Distutils/build_ext.py
+++ b/Cython/Distutils/build_ext.py
@@ -1,43 +1,130 @@
 import sys
+import os
 
-if 'setuptools' in sys.modules:
-    try:
-        from setuptools.command.build_ext import build_ext as _build_ext
-    except ImportError:
-        # We may be in the process of importing setuptools, which tries
-        # to import this.
-        from distutils.command.build_ext import build_ext as _build_ext
-else:
-    from distutils.command.build_ext import build_ext as _build_ext
+try:
+    from __builtin__ import basestring
+except ImportError:
+    basestring = str
+
+# Always inherit from the "build_ext" in distutils since setuptools already imports
+# it from Cython if available, and does the proper distutils fallback otherwise.
+# https://github.com/pypa/setuptools/blob/9f1822ee910df3df930a98ab99f66d18bb70659b/setuptools/command/build_ext.py#L16
 
+# setuptools imports Cython's "build_ext", so make sure we go first.
+_build_ext_module = sys.modules.get('setuptools.command.build_ext')
+if _build_ext_module is None:
+    import distutils.command.build_ext as _build_ext_module
+
+# setuptools remembers the original distutils "build_ext" as "_du_build_ext"
+_build_ext = getattr(_build_ext_module, '_du_build_ext', None)
+if _build_ext is None:
+    _build_ext = getattr(_build_ext_module, 'build_ext', None)
+if _build_ext is None:
+    from distutils.command.build_ext import build_ext as _build_ext
 
-class new_build_ext(_build_ext, object):
 
-    user_options = _build_ext.user_options[:]
-    boolean_options = _build_ext.boolean_options[:]
+class build_ext(_build_ext, object):
 
-    user_options.extend([
+    user_options = _build_ext.user_options + [
+        ('cython-cplus', None,
+             "generate C++ source files"),
+        ('cython-create-listing', None,
+             "write errors to a listing file"),
+        ('cython-line-directives', None,
+             "emit source line directives"),
+        ('cython-include-dirs=', None,
+             "path to the Cython include files" + _build_ext.sep_by),
         ('cython-c-in-temp', None,
              "put generated C files in temp directory"),
-        ])
+        ('cython-gen-pxi', None,
+            "generate .pxi file for public declarations"),
+        ('cython-directives=', None,
+            "compiler directive overrides"),
+        ('cython-gdb', None,
+             "generate debug information for cygdb"),
+        ('cython-compile-time-env', None,
+            "cython compile time environment"),
+        ]
 
-    boolean_options.extend([
-        'cython-c-in-temp'
-    ])
+    boolean_options = _build_ext.boolean_options + [
+        'cython-cplus', 'cython-create-listing', 'cython-line-directives',
+        'cython-c-in-temp', 'cython-gdb',
+    ]
 
     def initialize_options(self):
-        _build_ext.initialize_options(self)
+        super(build_ext, self).initialize_options()
+        self.cython_cplus = 0
+        self.cython_create_listing = 0
+        self.cython_line_directives = 0
+        self.cython_include_dirs = None
+        self.cython_directives = None
         self.cython_c_in_temp = 0
+        self.cython_gen_pxi = 0
+        self.cython_gdb = False
+        self.cython_compile_time_env = None
+
+    def finalize_options(self):
+        super(build_ext, self).finalize_options()
+        if self.cython_include_dirs is None:
+            self.cython_include_dirs = []
+        elif isinstance(self.cython_include_dirs, basestring):
+            self.cython_include_dirs = \
+                self.cython_include_dirs.split(os.pathsep)
+        if self.cython_directives is None:
+            self.cython_directives = {}
+
+    def get_extension_attr(self, extension, option_name, default=False):
+        return getattr(self, option_name) or getattr(extension, option_name, default)
 
     def build_extension(self, ext):
         from Cython.Build.Dependencies import cythonize
-        if self.cython_c_in_temp:
-            build_dir = self.build_temp
-        else:
-            build_dir = None
-        new_ext = cythonize(ext,force=self.force, quiet=self.verbose == 0, build_dir=build_dir)[0]
+
+        # Set up the include_path for the Cython compiler:
+        #    1.    Start with the command line option.
+        #    2.    Add in any (unique) paths from the extension
+        #        cython_include_dirs (if Cython.Distutils.extension is used).
+        #    3.    Add in any (unique) paths from the extension include_dirs
+        includes = list(self.cython_include_dirs)
+        for include_dir in getattr(ext, 'cython_include_dirs', []):
+            if include_dir not in includes:
+                includes.append(include_dir)
+
+        # In case extension.include_dirs is a generator, evaluate it and keep
+        # result
+        ext.include_dirs = list(ext.include_dirs)
+        for include_dir in ext.include_dirs + list(self.include_dirs):
+            if include_dir not in includes:
+                includes.append(include_dir)
+
+        # Set up Cython compiler directives:
+        #    1. Start with the command line option.
+        #    2. Add in any (unique) entries from the extension
+        #         cython_directives (if Cython.Distutils.extension is used).
+        directives = dict(self.cython_directives)
+        if hasattr(ext, "cython_directives"):
+            directives.update(ext.cython_directives)
+
+        if self.get_extension_attr(ext, 'cython_cplus'):
+            ext.language = 'c++'
+
+        options = {
+            'use_listing_file': self.get_extension_attr(ext, 'cython_create_listing'),
+            'emit_linenums': self.get_extension_attr(ext, 'cython_line_directives'),
+            'include_path': includes,
+            'compiler_directives': directives,
+            'build_dir': self.build_temp if self.get_extension_attr(ext, 'cython_c_in_temp') else None,
+            'generate_pxi': self.get_extension_attr(ext, 'cython_gen_pxi'),
+            'gdb_debug': self.get_extension_attr(ext, 'cython_gdb'),
+            'c_line_in_traceback': not getattr(ext, 'no_c_in_traceback', 0),
+            'compile_time_env': self.get_extension_attr(ext, 'cython_compile_time_env', default=None),
+        }
+
+        new_ext = cythonize(
+            ext,force=self.force, quiet=self.verbose == 0, **options
+        )[0]
+
         ext.sources = new_ext.sources
-        super(new_build_ext, self).build_extension(ext)
+        super(build_ext, self).build_extension(ext)
 
-# This will become new_build_ext in the future.
-from .old_build_ext import old_build_ext as build_ext
+# backward compatibility
+new_build_ext = build_ext
diff --git a/pyximport/pyxbuild.py b/pyximport/pyxbuild.py
--- a/pyximport/pyxbuild.py
+++ b/pyximport/pyxbuild.py
@@ -10,7 +10,7 @@
 from distutils.extension import Extension
 from distutils.util import grok_environment_error
 try:
-    from Cython.Distutils.build_ext import new_build_ext as build_ext
+    from Cython.Distutils.build_ext import build_ext
     HAS_CYTHON = True
 except ImportError:
     HAS_CYTHON = False
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -159,7 +159,7 @@ def compile_cython_modules(profile=False, coverage=False, compile_more=False, cy
         # XXX hack around setuptools quirk for '*.pyx' sources
         extensions[-1].sources[0] = pyx_source_file
 
-    from Cython.Distutils.build_ext import new_build_ext
+    from Cython.Distutils.build_ext import build_ext
     from Cython.Compiler.Options import get_directive_defaults
     get_directive_defaults().update(
         language_level=2,
@@ -175,7 +175,7 @@ def compile_cython_modules(profile=False, coverage=False, compile_more=False, cy
         sys.stderr.write("Enabled line tracing and profiling for the Cython binary modules\n")
 
     # not using cythonize() directly to let distutils decide whether building extensions was requested
-    add_command_class("build_ext", new_build_ext)
+    add_command_class("build_ext", build_ext)
     setup_args['ext_modules'] = extensions
 
 


Test patch:
diff --git a/tests/build/build_ext_cython_c_in_temp.srctree b/tests/build/build_ext_cython_c_in_temp.srctree
new file mode 100644
--- /dev/null
+++ b/tests/build/build_ext_cython_c_in_temp.srctree
@@ -0,0 +1,30 @@
+
+PYTHON setup.py build_ext --inplace --cython-c-in-temp
+PYTHON -c 'import mymodule; assert mymodule.test_string == "TEST"'
+PYTHON check_paths.py
+
+############# setup.py #############
+
+from Cython.Distutils.extension import Extension
+from Cython.Build import build_ext
+from distutils.core import setup
+
+setup(
+    name='Hello world app',
+    ext_modules = [
+        Extension(
+            name = 'mymodule',
+            sources=['mymodule.pyx'],
+        )
+    ],
+    cmdclass={'build_ext': build_ext},
+)
+
+######## mymodule.pyx ########
+
+test_string = "TEST"
+
+######## check_paths.py ########
+
+import os
+assert not os.path.exists("mymodule.c")
diff --git a/tests/build/build_ext_cython_cplus.srctree b/tests/build/build_ext_cython_cplus.srctree
new file mode 100644
--- /dev/null
+++ b/tests/build/build_ext_cython_cplus.srctree
@@ -0,0 +1,34 @@
+# tag: cpp
+
+PYTHON setup.py build_ext --inplace --cython-cplus
+PYTHON -c "import a; a.use_vector([1,2,3])"
+
+######## setup.py ########
+
+from Cython.Distutils.extension import Extension
+from Cython.Build import build_ext
+from distutils.core import setup
+
+setup(
+    name='Hello world app',
+    ext_modules = [
+        Extension(
+            name = 'a',
+            sources=['a.pyx'],
+        )
+    ],
+    cmdclass={'build_ext': build_ext},
+)
+
+######## a.pyx ########
+
+from libcpp.vector cimport vector
+
+def use_vector(L):
+    try:
+        v = new vector[int]()
+        for a in L:
+            v.push_back(a)
+        return v.size()
+    finally:
+        del v
diff --git a/tests/build/build_ext_cython_include_dirs.srctree b/tests/build/build_ext_cython_include_dirs.srctree
new file mode 100644
--- /dev/null
+++ b/tests/build/build_ext_cython_include_dirs.srctree
@@ -0,0 +1,50 @@
+
+PYTHON setup.py build_ext --inplace --cython-include-dirs=./headers1 --include-dirs=./headers2
+PYTHON -c 'import mymodule; assert mymodule.test_string == "TEST"; assert mymodule.header_value1 == 1; assert mymodule.header_value2 == 2; assert mymodule.header_value3 == 3; assert mymodule.header_value4 == 4'
+
+############# setup.py #############
+
+from Cython.Distutils.extension import Extension
+from Cython.Build import build_ext
+from distutils.core import setup
+
+setup(
+    name='Hello world app',
+    ext_modules = [
+        Extension(
+            name = 'mymodule',
+            sources=['mymodule.pyx'],
+            cython_include_dirs=['headers3'],
+            include_dirs=['./headers4']
+        )
+    ],
+    cmdclass={'build_ext': build_ext},
+)
+
+######## mymodule.pyx ########
+
+include "myheader1.pxi"
+include "myheader2.pxi"
+include "myheader3.pxi"
+include "myheader4.pxi"
+header_value1 = test_value1
+header_value2 = test_value2
+header_value3 = test_value3
+header_value4 = test_value4
+test_string = "TEST"
+
+######## headers1/myheader1.pxi ########
+
+cdef int test_value1 = 1
+
+######## headers2/myheader2.pxi ########
+
+cdef int test_value2 = 2
+
+######## headers3/myheader3.pxi ########
+
+cdef int test_value3 = 3
+
+######## headers4/myheader4.pxi ########
+
+cdef int test_value4 = 4