Repository: mesonbuild/meson

Issue:
Cython dependency handling in .pxd and .pxi files
**Describe the bug**

Cython has a notion of implementation source files (`.pyx`), declaration files (`.pxd`) and literal include files (`.pxi`). The distinction is similar to `.c` and `.h` files in C. The current syntax for tracking changes in declaration and include files is clumsy, especially for the cython->c++ transpile.

**To Reproduce**

In the current master branch, to specify a cython->c++ transile for a source with a `.pyx` and a `.pxd` file, `depend_files` argument to `custom_target` works (thanks to @rgommers for pointing this out). Here is an example where the implementation is split between `observable.pxd` and `observable.pyx` (https://github.com/ev-br/mc_lib/blob/master/mc_lib/meson.build):

```
# compile .cpp for the `observable` module
_observable_cpp = custom_target('observable_cpp',
  output : 'observable.cpp',
  input : 'observable.pyx',
  depend_files: 'observable.pxd',     # <---- this
  command : [cython, '--cplus', '-3', '--fast-fail', '@INPUT@', '-o', '@OUTPUT@']
)

# generate .so for observable
py3.extension_module(
  'observable', _observable_cpp,
  include_directories: inc_np,
  dependencies : [py3_dep],
  install: true,
  subdir: 'mc_lib'
)
```

With gh-9017, the syntax is greatly simplified:

```
# generate .so for the `observable` extension
py3.extension_module(
  'observable', 'observable.pyx',
  include_directories: inc_np,
  dependencies : py3_dep,
  override_options : ['cython_language=cpp'],   # <-----  no need for a custom_target, yay!
  install: true,
  subdir: 'mc_lib',
)
```

However, this does not track changes in `observable.pxd`. To reproduce the problem: compile once, introduce a syntax error into `observable.pxd`, try recompiling ( I did `$ meson install -C build`, but a direct ninja invocation would do) --- ninja does not see a syntax error:

```
ninja: Entering directory `/home/br/sweethome/proj/mc_lib/builddir'
ninja: no work to do.
```


**Expected behavior**

Expected behavior is that a change in a `pxd` or `pxi` file triggers a rebuild of a dependent extension module: similar to what happens for the `.c / .h` pairs. Either automatically, or via a simple `meson.build` syntax to declare a dependency.


**system parameters**

Native build on ubuntu focal, python 3.8.10, 
$ meson --version
0.59.99
$ ninja --version
1.10.2.git.kitware.jobserver-1




Discussion:
https://github.com/cython/cython/issues/1214

This was mentioned in the initial implementation of cython, and unfortunately meson has no way to guess what is needed without implementing a cython file parser. So we're blocked on upstream.
Since that cython issue mentions cythonize.py, I cannot help mentioning the comment in a variant of that tool (https://github.com/scipy/scipy/blob/master/tools/cythonize.py#L21): 
```
Simple script to invoke Cython ...; while waiting for a proper build system. Uses file hashes to figure out if rebuild is needed.
```
This comment dates back to at least 2012. We (at least some of us) are hoping that `meson` would be that proper build system we can use :-).

On a more serious note, two points:
- the OP cython example wraps a C++ header file, `_observable/observable.h`. Introducing a syntax error into the .h file, it's caught automatically somehow. 
- a simple syntax to mirror `depend_files` for `py3.extension_module` would be a great solution.
I'm not so sure depend_files would actually work... it would apply to the extension module, not to the pyx file, so it would not trigger a cython rebuild of observable.pyx ->  observable.cpp, it would just trigger a relink of all .o files.

As for the current behavior with observable.h

The syntax error in observable.h would be caught because C headers aren't exclusively used by cython, but also by GCC, and gcc emits depfiles and triggers rebuilds of observable.cpp -> observable.o when observable.h is modified.

Since observable.pyx does `cdef extern from "_observable/observable.h"` it *should* trigger a `cython` rebuild of observable.cpp, and since observable.cpp *also* does `#include "_observable/observable.h"` it *should* trigger a `gcc` rebuild of observable.o but...

Only the latter actually occurs. It may actually result in a build error even for valid changes to both .pyx and .h, because an old version of the .cpp file is compiled with the new version of the .h

Basically we need depfile support in cython, OR we need to add special syntax just for cython, which takes a dictionary of mappings rather than a simple list of files. 
> Basically we need depfile support in cython

That's https://github.com/cython/cython/issues/1459. Not much movement there, but the linked PR looks fairly straightforward (if incomplete, e.g. no handling of `.pxi` files).

> I'm not so sure depend_files would actually work... it would apply to the extension module, not to the pyx file

Naive question: in the absence of Cython depfile support, why would it be so bad to apply the `depend_files` to both the cython->C and C->.so step? It may do a little too much work sometimes, but that's better than doing too little.
It is the wrong technical solution, and highly inconvenient to users because any time you update the implementation files with new dependencies you need to manually sync meson.build too -- which forces a full buildsystem reconfigure.

And according to that linked PR I guess it should be easier to add the correct technical solution to Cython?
> And according to that linked PR I guess it should be easier to add the correct technical solution to Cython?

I hope so. One concern I have is that even if it gets implemented, it first goes in the Cython 3.0 branch (which is at least a year away) and the maintainer may not be willing to sync it back to `0.29.x` if it's a nontrivial change. But agreed, worth trying this first.
Depfile support has landed in the cython main branch.
> Depfile support has landed in the cython main branch.

Also 0.29.x branch now. (0.29.27, to be specific) 

Resolving patch:
diff --git a/mesonbuild/backend/ninjabackend.py b/mesonbuild/backend/ninjabackend.py
--- a/mesonbuild/backend/ninjabackend.py
+++ b/mesonbuild/backend/ninjabackend.py
@@ -1703,8 +1703,6 @@ def generate_cython_transpile(self, target: build.BuildTarget) -> \
         for src in target.get_sources():
             if src.endswith('.pyx'):
                 output = os.path.join(self.get_target_private_dir(target), f'{src}.{ext}')
-                args = args.copy()
-                args += cython.get_output_args(output)
                 element = NinjaBuildElement(
                     self.all_outputs, [output],
                     self.compiler_to_rule_name(cython),
@@ -1725,9 +1723,7 @@ def generate_cython_transpile(self, target: build.BuildTarget) -> \
                 else:
                     ssrc = os.path.join(gen.get_subdir(), ssrc)
                 if ssrc.endswith('.pyx'):
-                    args = args.copy()
                     output = os.path.join(self.get_target_private_dir(target), f'{ssrc}.{ext}')
-                    args += cython.get_output_args(output)
                     element = NinjaBuildElement(
                         self.all_outputs, [output],
                         self.compiler_to_rule_name(cython),
@@ -2235,9 +2231,18 @@ def generate_vala_compile_rules(self, compiler):
 
     def generate_cython_compile_rules(self, compiler: 'Compiler') -> None:
         rule = self.compiler_to_rule_name(compiler)
-        command = compiler.get_exelist() + ['$ARGS', '$in']
         description = 'Compiling Cython source $in'
-        self.add_rule(NinjaRule(rule, command, [], description, extra='restat = 1'))
+        command = compiler.get_exelist()
+
+        depargs = compiler.get_dependency_gen_args('$out', '$DEPFILE')
+        depfile = '$out.dep' if depargs else None
+
+        args = depargs + ['$ARGS', '$in']
+        args += NinjaCommandArg.list(compiler.get_output_args('$out'), Quoting.none)
+        self.add_rule(NinjaRule(rule, command + args, [],
+                                description,
+                                depfile=depfile,
+                                extra='restat = 1'))
 
     def generate_rust_compile_rules(self, compiler):
         rule = self.compiler_to_rule_name(compiler)
diff --git a/mesonbuild/compilers/cython.py b/mesonbuild/compilers/cython.py
--- a/mesonbuild/compilers/cython.py
+++ b/mesonbuild/compilers/cython.py
@@ -7,7 +7,7 @@
 import typing as T
 
 from .. import coredata
-from ..mesonlib import EnvironmentException, OptionKey
+from ..mesonlib import EnvironmentException, OptionKey, version_compare
 from .compilers import Compiler
 
 if T.TYPE_CHECKING:
@@ -40,6 +40,14 @@ def get_optimization_args(self, optimization_level: str) -> T.List[str]:
         # compiler might though
         return []
 
+    def get_dependency_gen_args(self, outtarget: str, outfile: str) -> T.List[str]:
+        if version_compare(self.version, '>=0.29.33'):
+            return ['-M']
+        return []
+
+    def get_depfile_suffix(self) -> str:
+        return 'dep'
+
     def sanity_check(self, work_dir: str, environment: 'Environment') -> None:
         code = 'print("hello world")'
         with self.cached_compile(code, environment.coredata) as p:
diff --git a/unittests/allplatformstests.py b/unittests/allplatformstests.py
--- a/unittests/allplatformstests.py
+++ b/unittests/allplatformstests.py
@@ -859,6 +859,27 @@ def test_build_generated_pyx_directly(self):
         name = "/".join(name)  # Glue list into a string
         self.build(target=name)
 
+    def test_build_pyx_depfiles(self):
+        # building regularly and then touching a depfile dependency should rebuild
+        testdir = os.path.join("test cases/cython", '2 generated sources')
+        env = get_fake_env(testdir, self.builddir, self.prefix)
+        try:
+            cython = detect_compiler_for(env, "cython", MachineChoice.HOST)
+            if not version_compare(cython.version, '>=0.29.33'):
+                raise SkipTest('Cython is too old')
+        except EnvironmentException:
+            raise SkipTest("Cython is not installed")
+        self.init(testdir)
+
+        targets = self.introspect('--targets')
+        for target in targets:
+            if target['name'].startswith('simpleinclude'):
+                name = target['name']
+        self.build()
+        self.utime(os.path.join(testdir, 'simplestuff.pxi'))
+        self.assertBuildRelinkedOnlyTarget(name)
+
+
     def test_internal_include_order(self):
         if mesonbuild.environment.detect_msys2_arch() and ('MESON_RSP_THRESHOLD' in os.environ):
             raise SkipTest('Test does not yet support gcc rsp files on msys2')


Test patch:
diff --git a/test cases/cython/2 generated sources/meson.build b/test cases/cython/2 generated sources/meson.build
--- a/test cases/cython/2 generated sources/meson.build	
+++ b/test cases/cython/2 generated sources/meson.build	
@@ -90,6 +90,11 @@ includestuff_ext = py3.extension_module(
   dependencies: stuff_pxi_dep
 )
 
+simpleinclude_ext = py3.extension_module(
+  'simpleinclude',
+  'simpleinclude.pyx',
+)
+
 subdir('libdir')
 
 test(
diff --git a/test cases/cython/2 generated sources/simpleinclude.pyx b/test cases/cython/2 generated sources/simpleinclude.pyx
new file mode 100644
--- /dev/null
+++ b/test cases/cython/2 generated sources/simpleinclude.pyx	
@@ -0,0 +1 @@
+include "simplestuff.pxi"
diff --git a/test cases/cython/2 generated sources/simplestuff.pxi b/test cases/cython/2 generated sources/simplestuff.pxi
new file mode 100644
--- /dev/null
+++ b/test cases/cython/2 generated sources/simplestuff.pxi	
@@ -0,0 +1,2 @@
+def func2():
+    print("Hello world")