Repository: cython/cython

Issue:
cythonize doesn’t recompile when only the pxd has changed in pure python mode
When a .py file is augmented by a .pxd one, cythonize won’t recompile the module when only the pxd has changed, or one of its dependencies, which often leads to broken code.

This doesn’t happen when the base module is a .pyx one.

Migrated from http://trac.cython.org/ticket/874



Discussion:
This is still true as of today (cython `3.0a1-79-g3de7a4b8f`):

```
$ ls
a.pxd  a.py  setup.py
```

`$ cat setup.py`
```py
from setuptools import setup
from Cython.Build import cythonize

setup(
    ext_modules = cythonize(["a.py"], language_level=3)
)
```

`$ cat a.py`
```py
def afunc(x):
    return x+2
```

`$ cat a.pxd`
```pyx
cdef int afunc(int x)
```

( first time compile -> ok )
```
$ python setup.py build_ext -i
Compiling a.py because it changed.
[1/1] Cythonizing a.py
running build_ext
building 'a' extension
creating build
creating build/temp.linux-x86_64-2.7
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-2.7.16=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -c a.c -o build/temp.linux-x86_64-2.7/a.o
creating build/lib.linux-x86_64-2.7
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-2.7.16=. -fstack-protector-strong -Wformat -Werror=format-security -Wl,-z,relro -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-2.7.16=. -fstack-protector-strong -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/a.o -o build/lib.linux-x86_64-2.7/a.so
copying build/lib.linux-x86_64-2.7/a.so -> 
```

( second time compile -> nothing is done (ok) )
```
$ python setup.py build_ext -i
running build_ext
copying build/lib.linux-x86_64-2.7/a.so -> 
```

( change `a.py` + compile -> `a.so` recompiled (ok) )
```
$ touch a.py 
$ python setup.py build_ext -i
Compiling a.py because it changed.
[1/1] Cythonizing a.py
running build_ext
building 'a' extension
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-2.7.16=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -c a.c -o build/temp.linux-x86_64-2.7/a.o
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-2.7.16=. -fstack-protector-strong -Wformat -Werror=format-security -Wl,-z,relro -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-2.7.16=. -fstack-protector-strong -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/a.o -o build/lib.linux-x86_64-2.7/a.so
copying build/lib.linux-x86_64-2.7/a.so -> 
```

( change `a.pxd`+ compile -> `a.so` is not recompiled (**WRONG**) )
```
$ touch a.pxd 
$ python setup.py build_ext -i
running build_ext
copying build/lib.linux-x86_64-2.7/a.so -> 
```

( `a.so` is not recompiled even if we change content of `a.pxd` (**WRONG**) )
```
$ echo ZZZ >a.pxd 
$ python setup.py build_ext -i
running build_ext
copying build/lib.linux-x86_64-2.7/a.so -> 
```

( it recompiles and gives the error only if `a.py` is changed )
```
$ touch a.py 
$ python setup.py build_ext -i
Compiling a.py because it changed.
[1/1] Cythonizing a.py

Error compiling Cython file:
------------------------------------------------------------
...
ZZZ
^
------------------------------------------------------------

a.pxd:1:0: undeclared name not builtin: ZZZ
Traceback (most recent call last):
  File "setup.py", line 5, in <module>
    ext_modules = cythonize(["a.py"], language_level=3)
  File "/home/kirr/src/tools/py/cython/Cython/Build/Dependencies.py", line 1105, in cythonize
    cythonize_one(*args)
  File "/home/kirr/src/tools/py/cython/Cython/Build/Dependencies.py", line 1263, in cythonize_one
    raise CompileError(None, pyx_file)
Cython.Compiler.Errors.CompileError: a.py
```

--------

This issue was hit for real with gevent where it was not evident why a code was crashing and only later it was discovered be a miscompilation issue: https://github.com/gevent/gevent/issues/1568#issuecomment-617432757 .

PR welcome.

Resolving patch:
diff --git a/Cython/Build/Dependencies.py b/Cython/Build/Dependencies.py
--- a/Cython/Build/Dependencies.py
+++ b/Cython/Build/Dependencies.py
@@ -534,7 +534,7 @@ def included_files(self, filename):
         for include in self.parse_dependencies(filename)[1]:
             include_path = join_path(os.path.dirname(filename), include)
             if not path_exists(include_path):
-                include_path = self.context.find_include_file(include, None)
+                include_path = self.context.find_include_file(include, source_file_path=filename)
             if include_path:
                 if '.' + os.path.sep in include_path:
                     include_path = os.path.normpath(include_path)
@@ -588,17 +588,18 @@ def find_pxd(self, module, filename=None):
                     return None   # FIXME: error?
                 module_path.pop(0)
             relative = '.'.join(package_path + module_path)
-            pxd = self.context.find_pxd_file(relative, None)
+            pxd = self.context.find_pxd_file(relative, source_file_path=filename)
             if pxd:
                 return pxd
         if is_relative:
             return None   # FIXME: error?
-        return self.context.find_pxd_file(module, None)
+        return self.context.find_pxd_file(module, source_file_path=filename)
 
     @cached_method
     def cimported_files(self, filename):
-        if filename[-4:] == '.pyx' and path_exists(filename[:-4] + '.pxd'):
-            pxd_list = [filename[:-4] + '.pxd']
+        filename_root, filename_ext = os.path.splitext(filename)
+        if filename_ext in ('.pyx', '.py') and path_exists(filename_root + '.pxd'):
+            pxd_list = [filename_root + '.pxd']
         else:
             pxd_list = []
         # Cimports generates all possible combinations package.module


Test patch:
diff --git a/Cython/Build/Tests/TestRecythonize.py b/Cython/Build/Tests/TestRecythonize.py
new file mode 100644
--- /dev/null
+++ b/Cython/Build/Tests/TestRecythonize.py
@@ -0,0 +1,212 @@
+import shutil
+import os
+import tempfile
+import time
+
+import Cython.Build.Dependencies
+import Cython.Utils
+from Cython.TestUtils import CythonTest
+
+
+def fresh_cythonize(*args, **kwargs):
+    Cython.Utils.clear_function_caches()
+    Cython.Build.Dependencies._dep_tree = None  # discard method caches
+    Cython.Build.Dependencies.cythonize(*args, **kwargs)
+
+class TestRecythonize(CythonTest):
+
+    def setUp(self):
+        CythonTest.setUp(self)
+        self.temp_dir = (
+            tempfile.mkdtemp(
+                prefix='recythonize-test',
+                dir='TEST_TMP' if os.path.isdir('TEST_TMP') else None
+            )
+        )
+
+    def tearDown(self):
+        CythonTest.tearDown(self)
+        shutil.rmtree(self.temp_dir)
+
+    def test_recythonize_pyx_on_pxd_change(self):
+
+        src_dir = tempfile.mkdtemp(prefix='src', dir=self.temp_dir)
+
+        a_pxd = os.path.join(src_dir, 'a.pxd')
+        a_pyx = os.path.join(src_dir, 'a.pyx')
+        a_c = os.path.join(src_dir, 'a.c')
+        dep_tree = Cython.Build.Dependencies.create_dependency_tree()
+
+        with open(a_pxd, 'w') as f:
+            f.write('cdef int value\n')
+
+        with open(a_pyx, 'w') as f:
+            f.write('value = 1\n')
+
+
+        # The dependencies for "a.pyx" are "a.pxd" and "a.pyx".
+        self.assertEqual({a_pxd, a_pyx}, dep_tree.all_dependencies(a_pyx))
+
+        # Cythonize to create a.c
+        fresh_cythonize(a_pyx)
+
+        # Sleep to address coarse time-stamp precision.
+        time.sleep(1)
+
+        with open(a_c) as f:
+            a_c_contents1 = f.read()
+
+        with open(a_pxd, 'w') as f:
+            f.write('cdef double value\n')
+
+        fresh_cythonize(a_pyx)
+
+        with open(a_c) as f:
+            a_c_contents2 = f.read()
+
+        self.assertTrue("__pyx_v_1a_value = 1;" in a_c_contents1)
+        self.assertFalse("__pyx_v_1a_value = 1;" in a_c_contents2)
+        self.assertTrue("__pyx_v_1a_value = 1.0;" in a_c_contents2)
+        self.assertFalse("__pyx_v_1a_value = 1.0;" in a_c_contents1)
+
+
+    def test_recythonize_py_on_pxd_change(self):
+
+        src_dir = tempfile.mkdtemp(prefix='src', dir=self.temp_dir)
+
+        a_pxd = os.path.join(src_dir, 'a.pxd')
+        a_py = os.path.join(src_dir, 'a.py')
+        a_c = os.path.join(src_dir, 'a.c')
+        dep_tree = Cython.Build.Dependencies.create_dependency_tree()
+
+        with open(a_pxd, 'w') as f:
+            f.write('cdef int value\n')
+
+        with open(a_py, 'w') as f:
+            f.write('value = 1\n')
+
+
+        # The dependencies for "a.py" are "a.pxd" and "a.py".
+        self.assertEqual({a_pxd, a_py}, dep_tree.all_dependencies(a_py))
+
+        # Cythonize to create a.c
+        fresh_cythonize(a_py)
+
+        # Sleep to address coarse time-stamp precision.
+        time.sleep(1)
+
+        with open(a_c) as f:
+            a_c_contents1 = f.read()
+
+        with open(a_pxd, 'w') as f:
+            f.write('cdef double value\n')
+
+        fresh_cythonize(a_py)
+
+        with open(a_c) as f:
+            a_c_contents2 = f.read()
+
+
+        self.assertTrue("__pyx_v_1a_value = 1;" in a_c_contents1)
+        self.assertFalse("__pyx_v_1a_value = 1;" in a_c_contents2)
+        self.assertTrue("__pyx_v_1a_value = 1.0;" in a_c_contents2)
+        self.assertFalse("__pyx_v_1a_value = 1.0;" in a_c_contents1)
+
+    def test_recythonize_pyx_on_dep_pxd_change(self):
+        src_dir = tempfile.mkdtemp(prefix='src', dir=self.temp_dir)
+
+        a_pxd = os.path.join(src_dir, 'a.pxd')
+        a_pyx = os.path.join(src_dir, 'a.pyx')
+        b_pyx = os.path.join(src_dir, 'b.pyx')
+        b_c = os.path.join(src_dir, 'b.c')
+        dep_tree = Cython.Build.Dependencies.create_dependency_tree()
+
+        with open(a_pxd, 'w') as f:
+            f.write('cdef int value\n')
+
+        with open(a_pyx, 'w') as f:
+            f.write('value = 1\n')
+
+        with open(b_pyx, 'w') as f:
+            f.write('cimport a\n' + 'a.value = 2\n')
+
+
+        # The dependencies for "b.pyx" are "a.pxd" and "b.pyx".
+        self.assertEqual({a_pxd, b_pyx}, dep_tree.all_dependencies(b_pyx))
+
+
+        # Cythonize to create b.c
+        fresh_cythonize([a_pyx, b_pyx])
+
+        # Sleep to address coarse time-stamp precision.
+        time.sleep(1)
+
+        with open(b_c) as f:
+            b_c_contents1 = f.read()
+
+        with open(a_pxd, 'w') as f:
+            f.write('cdef double value\n')
+
+        fresh_cythonize([a_pyx, b_pyx])
+
+        with open(b_c) as f:
+            b_c_contents2 = f.read()
+
+
+
+        self.assertTrue("__pyx_v_1a_value = 2;" in b_c_contents1)
+        self.assertFalse("__pyx_v_1a_value = 2;" in b_c_contents2)
+        self.assertTrue("__pyx_v_1a_value = 2.0;" in b_c_contents2)
+        self.assertFalse("__pyx_v_1a_value = 2.0;" in b_c_contents1)
+
+
+
+    def test_recythonize_py_on_dep_pxd_change(self):
+
+        src_dir = tempfile.mkdtemp(prefix='src', dir=self.temp_dir)
+
+        a_pxd = os.path.join(src_dir, 'a.pxd')
+        a_pyx = os.path.join(src_dir, 'a.pyx')
+        b_pxd = os.path.join(src_dir, 'b.pxd')
+        b_py = os.path.join(src_dir, 'b.py')
+        b_c = os.path.join(src_dir, 'b.c')
+        dep_tree = Cython.Build.Dependencies.create_dependency_tree()
+
+        with open(a_pxd, 'w') as f:
+            f.write('cdef int value\n')
+
+        with open(a_pyx, 'w') as f:
+            f.write('value = 1\n')
+
+        with open(b_pxd, 'w') as f:
+            f.write('cimport a\n')
+
+        with open(b_py, 'w') as f:
+            f.write('a.value = 2\n')
+
+
+        # The dependencies for b.py are "a.pxd", "b.pxd" and "b.py".
+        self.assertEqual({a_pxd, b_pxd, b_py}, dep_tree.all_dependencies(b_py))
+
+
+        # Cythonize to create b.c
+        fresh_cythonize([a_pyx, b_py])
+
+        # Sleep to address coarse time-stamp precision.
+        time.sleep(1)
+
+        with open(b_c) as f:
+            b_c_contents1 = f.read()
+
+        with open(a_pxd, 'w') as f:
+            f.write('cdef double value\n')
+
+        fresh_cythonize([a_pyx, b_py])
+
+        with open(b_c) as f:
+            b_c_contents2 = f.read()
+
+        self.assertTrue("__pyx_v_1a_value = 2;" in b_c_contents1)
+        self.assertFalse("__pyx_v_1a_value = 2;" in b_c_contents2)
+        self.assertTrue("__pyx_v_1a_value = 2.0;" in b_c_contents2)
+        self.assertFalse("__pyx_v_1a_value = 2.0;" in b_c_contents1)