Repository: cython/cython
Pull request: Make 'depfile' generation recognise 'cython.cimports.*' dependencies and Unicode names
Fixes https://github.com/cython/cython/issues/6938

Problem statement:
[BUG] PXD dependencies are not listed in the depfiles for pure python code.
### Describe the bug

If you have two equivalent cython files, one in cython and one in pure python, only the cython version will output correct dependency information when compiled with `cython -M`. This happens even when including pxd files outside the standard library.

### Code to reproduce the behaviour:

```python
# File: test.py
from cython.cimports.libc.stdint import uint32_t

def foo(x: uint32_t):
    return x
```
```cython
# File: test.pyx
from libc.stdint cimport uint32_t

def foo(uint32_t x):
    return x
```

Then compile with:
```sh
$ cython -3 test.py -M -o test.py.c
$ cython -3 test.pyx -M -o test.pyx.c
```

Inspecting the resulting depfiles gives:
```sh
$ cat test.pyx.c.dep
test.pyx.c: \
  test.pyx \
  ../../.local/lib/python3.10/site-packages/Cython/Includes/libc/stdint.pxd
$ cat test.py.c.dep
test.py.c: \
  test.py
```

### Expected behaviour

The two dependency files should be the same.

### OS

Ubuntu 22.04

### Python version

3.10.12

### Cython version

3.1.1

### Additional context

_No response_

Hints from issue discussion:
Thanks for the report. The depfile generation hadn't caught up with the features yet, it seems.

Commit message: Fix test on Windows.

Patch to review:
diff --git a/Cython/Build/Dependencies.py b/Cython/Build/Dependencies.py
index 477ff586fee..5714f6b6537 100644
--- a/Cython/Build/Dependencies.py
+++ b/Cython/Build/Dependencies.py
@@ -392,13 +392,17 @@ def parse_code(start: cython.Py_ssize_t, in_fstring: cython.bint = False) -> cyt
 # We need to allow spaces to allow for conditional compilation like
 # IF ...:
 #     cimport ...
-dependency_regex = re.compile(r"(?:^\s*from +([0-9a-zA-Z_.]+) +cimport)|"
-                              r"(?:^\s*cimport +([0-9a-zA-Z_.]+(?: *, *[0-9a-zA-Z_.]+)*))|"
-                              r"(?:^\s*cdef +extern +from +['\"]([^'\"]+)['\"])|"
-                              r"(?:^\s*include +['\"]([^'\"]+)['\"])", re.M)
+dependency_regex = re.compile(
+    r"(?:^\s*from +cython\.cimports\.([\w.]+) +c?import)|"
+    r"(?:^\s*from +([\w.]+) +cimport)|"
+    r"(?:^\s*c?import +cython\.cimports\.([\w.]+(?: *, *[\w.]+)*))|"
+    r"(?:^\s*cimport +([\w.]+(?: *, *[\w.]+)*))|"
+    r"(?:^\s*cdef +extern +from +['\"]([^'\"]+)['\"])|"
+    r"(?:^\s*include +['\"]([^'\"]+)['\"])",
+    re.M)
 dependency_after_from_regex = re.compile(
-    r"(?:^\s+\(([0-9a-zA-Z_., ]*)\)[#\n])|"
-    r"(?:^\s+([0-9a-zA-Z_., ]*)[#\n])",
+    r"(?:^\s+\(([\w., ]*)\)[#\n])|"
+    r"(?:^\s+([\w., ]*)[#\n])",
     re.M)
 
 
@@ -486,7 +490,12 @@ def parse_dependencies(source_filename):
     includes = []
     externs  = []
     for m in dependency_regex.finditer(source):
-        cimport_from, cimport_list, extern, include = m.groups()
+        pycimports_from, cimport_from, pycimports_list, cimport_list, extern, include = m.groups()
+        if pycimports_from:
+            cimport_from = pycimports_from
+        if pycimports_list:
+            cimport_list = pycimports_list
+
         if cimport_from:
             cimports.append(cimport_from)
             m_after_from = dependency_after_from_regex.search(source, pos=m.end())
diff --git a/tests/build/depfile.srctree b/tests/build/depfile.srctree
index 0b8019df957..02dd6981f66 100644
--- a/tests/build/depfile.srctree
+++ b/tests/build/depfile.srctree
@@ -15,9 +15,12 @@ print(foo())
 
 ######## baz.pxi ########
 
+from cython.cimports.libc.math import sin
+
 def foo():
     return "foo"
 
+print(sin(0))
 
 ######## bar.pxd ########
 
@@ -28,6 +31,7 @@ cdef inline void empty():
 ######## check.py ########
 
 with open("foo.c.dep", "r") as f:
-    contents = f.read().replace("\\\n", " ").replace("\n", " ")
+    contents = f.read().replace("\\\n", " ").replace('\\', '/').replace('../', '')
 
-assert sorted(contents.split()) == ['bar.pxd', 'baz.pxi', 'foo.c:', 'foo.pyx'], contents
+assert sorted(contents.split()) == [
+    'Cython/Includes/libc/math.pxd', 'bar.pxd', 'baz.pxi', 'foo.c:', 'foo.pyx'], contents


Code review comments:
diff_hunk: @@ -392,13 +392,17 @@ def parse_code(start: cython.Py_ssize_t, in_fstring: cython.bint = False) -> cyt
 # We need to allow spaces to allow for conditional compilation like
 # IF ...:
 #     cimport ...
-dependency_regex = re.compile(r"(?:^\s*from +([0-9a-zA-Z_.]+) +cimport)|"
-                              r"(?:^\s*cimport +([0-9a-zA-Z_.]+(?: *, *[0-9a-zA-Z_.]+)*))|"
-                              r"(?:^\s*cdef +extern +from +['\"]([^'\"]+)['\"])|"
-                              r"(?:^\s*include +['\"]([^'\"]+)['\"])", re.M)
+dependency_regex = re.compile(
+    r"(?:^\s*from +cython\.cimports\.([\w.]+) +c?import)|"
path: Cython/Build/Dependencies.py
text: @user1:
I think some of the ` +` should be `\s+`. Python definitely seems to accept tabs in the middle of imports (e.g. `exec("from\tmath import\t\tsin")` works for me)

@author:
We're parsing across lines here, I guess that was the reason why the code didn't use `\s` (which includes newline). But I don't think a newline can realistically appear before the import name list in the end. I'll add `\s` at least in a few places.

@user1:
Ah I forgot about newlines. Looking at https://docs.python.org/3/reference/lexical_analysis.html it should probably be `[ \t\f]` then.