Repository: cython/cython

Issue:
Coverage plugin is broken on Cython modules in packages with sources in src-layout
_(Ref: https://twitter.com/webKnjaZ/status/1265373614003625987)_

Having a module under src-layout causes `Cython.Coverage.Plugin` to not set itself as a tracer in the DB. And even when it's set, it fails to matchthe module with its sources

I've made a reproducer repo with GitHub Actions Workflow integrated as a demo of this bug:
- https://github.com/webknjaz/cython-coverage-extern-bug-repro-issue3636
- https://github.com/webknjaz/cython-coverage-extern-bug-repro-issue3636/runs/718112079?check_suite_focus=true#step:13:6

When this happens, the following message is printed out during the report stage:
```console
Couldn't parse '..../src/call_mymath/call_mymath.pyx' as Python source: 'invalid syntax' at line 1
```

Reproducer details:
==================
It's based on the example https://github.com/cython/cython/tree/master/Demos/libraries with a slightly modified structure:
```console
|- mymath.c
|- mymath.h
|- setup.py
|- src/
    |- call_mymath/
        |- call_mymath.pyx
        |- __init__.py
    |-flat_ns_call_mymath.pyx
|- test_cython.py
```
`flat_ns_call_mymath.pyx` is the same as `call_mymath.pyx` (the only thing modified inside is relative path to the C-header).

`setup.py` puts both `flat_ns_call_mymath.so` and `call_mymath.so` under `call_mymath` package in `site-packages`. So they are importable as `call_mymath.flat_ns_call_mymath` and `call_mymath.call_mymath`. `call_mymath.flat_ns_call_mymath` shows up in the report and `call_mymath.call_mymath` doesn't because `Cython.Coverage.Plugin` fails to map it with the source and claim the file tracing rights.


STR
===
1. `pip install coverage Cython`
2. `CFLAGS=-DCYTHON_TRACE=1 cythonize --inplace --directive=embedsignature=True --directive=emit_code_comments=True --directive=linetrace=True --directive=profile=True '**/*.pyx'`
3. `pip install .`
4. `coverage run -m test_cython`
5. `coverage report` -- this is where it fails with
    ```console
    Couldn't parse '..../src/call_mymath/call_mymath.pyx' as Python source: 'invalid syntax' at line 1
    ```
    (adding `-i` makes at least `flat_ns_call_mymath` show up correctly)

When looking at `.coverage` using `sqlite3`, `flat_ns_call_mymath.pyx` is mapped to `Cython.Coverage.Plugin` in the `tracer` table while there's no record for `call_mymath.pyx` there.
Coverage plugin is broken on Cython modules in packages with sources in src-layout
_(Ref: https://twitter.com/webKnjaZ/status/1265373614003625987)_

Having a module under src-layout causes `Cython.Coverage.Plugin` to not set itself as a tracer in the DB. And even when it's set, it fails to matchthe module with its sources

I've made a reproducer repo with GitHub Actions Workflow integrated as a demo of this bug:
- https://github.com/webknjaz/cython-coverage-extern-bug-repro-issue3636
- https://github.com/webknjaz/cython-coverage-extern-bug-repro-issue3636/runs/718112079?check_suite_focus=true#step:13:6

When this happens, the following message is printed out during the report stage:
```console
Couldn't parse '..../src/call_mymath/call_mymath.pyx' as Python source: 'invalid syntax' at line 1
```

Reproducer details:
==================
It's based on the example https://github.com/cython/cython/tree/master/Demos/libraries with a slightly modified structure:
```console
|- mymath.c
|- mymath.h
|- setup.py
|- src/
    |- call_mymath/
        |- call_mymath.pyx
        |- __init__.py
    |-flat_ns_call_mymath.pyx
|- test_cython.py
```
`flat_ns_call_mymath.pyx` is the same as `call_mymath.pyx` (the only thing modified inside is relative path to the C-header).

`setup.py` puts both `flat_ns_call_mymath.so` and `call_mymath.so` under `call_mymath` package in `site-packages`. So they are importable as `call_mymath.flat_ns_call_mymath` and `call_mymath.call_mymath`. `call_mymath.flat_ns_call_mymath` shows up in the report and `call_mymath.call_mymath` doesn't because `Cython.Coverage.Plugin` fails to map it with the source and claim the file tracing rights.


STR
===
1. `pip install coverage Cython`
2. `CFLAGS=-DCYTHON_TRACE=1 cythonize --inplace --directive=embedsignature=True --directive=emit_code_comments=True --directive=linetrace=True --directive=profile=True '**/*.pyx'`
3. `pip install .`
4. `coverage run -m test_cython`
5. `coverage report` -- this is where it fails with
    ```console
    Couldn't parse '..../src/call_mymath/call_mymath.pyx' as Python source: 'invalid syntax' at line 1
    ```
    (adding `-i` makes at least `flat_ns_call_mymath` show up correctly)

When looking at `.coverage` using `sqlite3`, `flat_ns_call_mymath.pyx` is mapped to `Cython.Coverage.Plugin` in the `tracer` table while there's no record for `call_mymath.pyx` there.


Discussion:
So here's the reproducer: https://github.com/webknjaz/cython-coverage-extern-bug-repro-issue3636/runs/718112079?check_suite_focus=true#step:13:6.

@scoder ^
Turns out that Cythonize generates C-sources that don't include `src/` prefix in path comments.

Just adding those comments there makes it show up in the log:
```console
$ sed -i s#/\\*[[:space:]]'"'call_mymath/#/*\ '"'src/call_mymath/#g src/call_mymath/call_mymath.c
``` 

```diff
--- src/call_mymath/call_mymath.c.orig	2020-05-28 23:15:32.663441119 +0200
+++ src/call_mymath/call_mymath.c	2020-05-28 23:17:46.511099303 +0200
@@ -1243,7 +1243,7 @@
 static PyObject *__pyx_tuple__2;
 /* Late includes */
 
-/* "call_mymath/call_mymath.pyx":4
+/* "src/call_mymath/call_mymath.pyx":4
  *     double sinc(double)
  * 
  * def call_sinc(x):             # <<<<<<<<<<<<<<
@@ -1278,7 +1278,7 @@
   __Pyx_RefNannySetupContext("call_sinc", 0);
   __Pyx_TraceCall("call_sinc", __pyx_f[0], 4, 0, __PYX_ERR(0, 4, __pyx_L1_error));
 
-  /* "call_mymath/call_mymath.pyx":5
+  /* "src/call_mymath/call_mymath.pyx":5
  * 
  * def call_sinc(x):
  *     return sinc(x)             # <<<<<<<<<<<<<<
@@ -1292,7 +1292,7 @@
   __pyx_t_2 = 0;
   goto __pyx_L0;
 
-  /* "call_mymath/call_mymath.pyx":4
+  /* "src/call_mymath/call_mymath.pyx":4
  *     double sinc(double)
  * 
  * def call_sinc(x):             # <<<<<<<<<<<<<<
@@ -1375,7 +1375,7 @@
   __Pyx_RefNannyDeclarations
   __Pyx_RefNannySetupContext("__Pyx_InitCachedConstants", 0);
 
-  /* "call_mymath/call_mymath.pyx":4
+  /* "src/call_mymath/call_mymath.pyx":4
  *     double sinc(double)
  * 
  * def call_sinc(x):             # <<<<<<<<<<<<<<
@@ -1667,7 +1667,7 @@
   #endif
   __Pyx_TraceCall("__Pyx_PyMODINIT_FUNC PyInit_call_mymath(void)", __pyx_f[0], 1, 0, __PYX_ERR(0, 1, __pyx_L1_error));
 
-  /* "call_mymath/call_mymath.pyx":4
+  /* "src/call_mymath/call_mymath.pyx":4
  *     double sinc(double)
  * 
  * def call_sinc(x):             # <<<<<<<<<<<<<<
@@ -1679,7 +1679,7 @@
   if (PyDict_SetItem(__pyx_d, __pyx_n_s_call_sinc, __pyx_t_1) < 0) __PYX_ERR(0, 4, __pyx_L1_error)
   __Pyx_DECREF(__pyx_t_1); __pyx_t_1 = 0;
 
-  /* "call_mymath/call_mymath.pyx":1
+  /* "src/call_mymath/call_mymath.pyx":1
  * cdef extern from "../../mymath.h":             # <<<<<<<<<<<<<<
  *     double sinc(double)
  * 
```

But this doesn't explain why `flat_ns_call_mymath.pyx` works. If both don't have `src/` both should work the same way but they don't...
So when `Cython.Coverage._find_dep_file_path(main_file='/home/wk/src/github/webknjaz/cython-coverage-extern-bug-repro-issue3636/src/call_mymath/call_mymath.c', file_path='call_mymath/call_mymath.pyx', relative_path_search=True)` is called, it returns `/home/wk/src/github/webknjaz/cython-coverage-extern-bug-repro-issue3636/call_mymath/call_mymath.pyx` with `src/` missing because it doesn't do any exit checks.

I'm currently confused by the design decisions: should C-files contain a full path relative to the root of the project or `_find_dep_file_path()` should guess the source correctly?
So here's the reproducer: https://github.com/webknjaz/cython-coverage-extern-bug-repro-issue3636/runs/718112079?check_suite_focus=true#step:13:6.

@scoder ^
Turns out that Cythonize generates C-sources that don't include `src/` prefix in path comments.

Just adding those comments there makes it show up in the log:
```console
$ sed -i s#/\\*[[:space:]]'"'call_mymath/#/*\ '"'src/call_mymath/#g src/call_mymath/call_mymath.c
``` 

```diff
--- src/call_mymath/call_mymath.c.orig	2020-05-28 23:15:32.663441119 +0200
+++ src/call_mymath/call_mymath.c	2020-05-28 23:17:46.511099303 +0200
@@ -1243,7 +1243,7 @@
 static PyObject *__pyx_tuple__2;
 /* Late includes */
 
-/* "call_mymath/call_mymath.pyx":4
+/* "src/call_mymath/call_mymath.pyx":4
  *     double sinc(double)
  * 
  * def call_sinc(x):             # <<<<<<<<<<<<<<
@@ -1278,7 +1278,7 @@
   __Pyx_RefNannySetupContext("call_sinc", 0);
   __Pyx_TraceCall("call_sinc", __pyx_f[0], 4, 0, __PYX_ERR(0, 4, __pyx_L1_error));
 
-  /* "call_mymath/call_mymath.pyx":5
+  /* "src/call_mymath/call_mymath.pyx":5
  * 
  * def call_sinc(x):
  *     return sinc(x)             # <<<<<<<<<<<<<<
@@ -1292,7 +1292,7 @@
   __pyx_t_2 = 0;
   goto __pyx_L0;
 
-  /* "call_mymath/call_mymath.pyx":4
+  /* "src/call_mymath/call_mymath.pyx":4
  *     double sinc(double)
  * 
  * def call_sinc(x):             # <<<<<<<<<<<<<<
@@ -1375,7 +1375,7 @@
   __Pyx_RefNannyDeclarations
   __Pyx_RefNannySetupContext("__Pyx_InitCachedConstants", 0);
 
-  /* "call_mymath/call_mymath.pyx":4
+  /* "src/call_mymath/call_mymath.pyx":4
  *     double sinc(double)
  * 
  * def call_sinc(x):             # <<<<<<<<<<<<<<
@@ -1667,7 +1667,7 @@
   #endif
   __Pyx_TraceCall("__Pyx_PyMODINIT_FUNC PyInit_call_mymath(void)", __pyx_f[0], 1, 0, __PYX_ERR(0, 1, __pyx_L1_error));
 
-  /* "call_mymath/call_mymath.pyx":4
+  /* "src/call_mymath/call_mymath.pyx":4
  *     double sinc(double)
  * 
  * def call_sinc(x):             # <<<<<<<<<<<<<<
@@ -1679,7 +1679,7 @@
   if (PyDict_SetItem(__pyx_d, __pyx_n_s_call_sinc, __pyx_t_1) < 0) __PYX_ERR(0, 4, __pyx_L1_error)
   __Pyx_DECREF(__pyx_t_1); __pyx_t_1 = 0;
 
-  /* "call_mymath/call_mymath.pyx":1
+  /* "src/call_mymath/call_mymath.pyx":1
  * cdef extern from "../../mymath.h":             # <<<<<<<<<<<<<<
  *     double sinc(double)
  * 
```

But this doesn't explain why `flat_ns_call_mymath.pyx` works. If both don't have `src/` both should work the same way but they don't...
So when `Cython.Coverage._find_dep_file_path(main_file='/home/wk/src/github/webknjaz/cython-coverage-extern-bug-repro-issue3636/src/call_mymath/call_mymath.c', file_path='call_mymath/call_mymath.pyx', relative_path_search=True)` is called, it returns `/home/wk/src/github/webknjaz/cython-coverage-extern-bug-repro-issue3636/call_mymath/call_mymath.pyx` with `src/` missing because it doesn't do any exit checks.

I'm currently confused by the design decisions: should C-files contain a full path relative to the root of the project or `_find_dep_file_path()` should guess the source correctly?

Resolving patch:
diff --git a/Cython/Coverage.py b/Cython/Coverage.py
--- a/Cython/Coverage.py
+++ b/Cython/Coverage.py
@@ -83,6 +83,23 @@ def _find_dep_file_path(main_file, file_path, relative_path_search=False):
         rel_file_path = os.path.join(os.path.dirname(main_file), file_path)
         if os.path.exists(rel_file_path):
             abs_path = os.path.abspath(rel_file_path)
+
+        abs_no_ext = os.path.splitext(abs_path)[0]
+        file_no_ext, extension = os.path.splitext(file_path)
+        # We check if the paths match by matching the directories in reverse order.
+        # pkg/module.pyx /long/absolute_path/bla/bla/site-packages/pkg/module.c should match.
+        # this will match the pairs: module-module and pkg-pkg. After which there is nothing left to zip.
+        abs_no_ext = os.path.normpath(abs_no_ext)
+        file_no_ext = os.path.normpath(file_no_ext)
+        matching_paths = zip(reversed(abs_no_ext.split(os.sep)), reversed(file_no_ext.split(os.sep)))
+        for one, other in matching_paths:
+            if one != other:
+                break
+        else:  # No mismatches detected
+            matching_abs_path = os.path.splitext(main_file)[0] + extension
+            if os.path.exists(matching_abs_path):
+                return canonical_filename(matching_abs_path)
+
     # search sys.path for external locations if a valid file hasn't been found
     if not os.path.exists(abs_path):
         for sys_path in sys.path:


Test patch:
diff --git a/tests/run/coverage_cmd_src_pkg_layout.srctree b/tests/run/coverage_cmd_src_pkg_layout.srctree
new file mode 100644
--- /dev/null
+++ b/tests/run/coverage_cmd_src_pkg_layout.srctree
@@ -0,0 +1,177 @@
+# mode: run
+# tag: coverage,trace
+
+"""
+PYTHON -m pip install .
+PYTHON setup.py build_ext --inplace
+PYTHON -m coverage run --source=pkg coverage_test.py
+PYTHON collect_coverage.py
+"""
+
+######## setup.py ########
+
+from setuptools import Extension, find_packages, setup
+from Cython.Build import cythonize
+
+MODULES = [
+        Extension("pkg.module1", ["src/pkg/module1.pyx"]),
+        ]
+
+setup(
+    name="pkg",
+    zip_safe=False,
+    packages=find_packages('src'),
+    package_data={'pkg': ['*.pxd', '*.pyx']},
+    package_dir={'': 'src'},
+    ext_modules= cythonize(MODULES)
+    )
+
+
+######## .coveragerc ########
+[run]
+plugins = Cython.Coverage
+
+######## src/pkg/__init__.py ########
+
+######## src/pkg/module1.pyx ########
+# cython: linetrace=True
+# distutils: define_macros=CYTHON_TRACE=1
+
+def func1(int a, int b):
+    cdef int x = 1      #  5
+    c = func2(a) + b    #  6
+    return x + c        #  7
+
+
+def func2(int a):
+    return a * 2       # 11
+
+######## coverage_test.py ########
+
+import os.path
+from pkg import module1
+
+
+assert not any(
+    module1.__file__.endswith(ext)
+    for ext in '.py .pyc .pyo .pyw .pyx .pxi'.split()
+), module.__file__
+
+
+def run_coverage(module):
+    assert module.func1(1, 2) == (1 * 2) + 2 + 1
+    assert module.func2(2) == 2 * 2
+
+
+if __name__ == '__main__':
+    run_coverage(module1)
+
+
+######## collect_coverage.py ########
+
+import re
+import sys
+import os
+import os.path
+import subprocess
+from glob import iglob
+
+
+def run_coverage_command(*command):
+    env = dict(os.environ, LANG='', LC_ALL='C')
+    process = subprocess.Popen(
+        [sys.executable, '-m', 'coverage'] + list(command),
+        stdout=subprocess.PIPE, env=env)
+    stdout, _ = process.communicate()
+    return stdout
+
+
+def run_report():
+    stdout = run_coverage_command('report', '--show-missing')
+    stdout = stdout.decode('iso8859-1')  # 'safe' decoding
+    lines = stdout.splitlines()
+    print(stdout)
+
+    module_path = 'module1.pyx'
+    assert any(module_path in line for line in lines), (
+        "'%s' not found in coverage report:\n\n%s" % (module_path, stdout))
+
+    files = {}
+    line_iter = iter(lines)
+    for line in line_iter:
+        if line.startswith('---'):
+            break
+    extend = [''] * 2
+    for line in line_iter:
+        if not line or line.startswith('---'):
+            continue
+        name, statements, missed, covered, _missing = (line.split(None, 4) + extend)[:5]
+        missing = []
+        for start, end in re.findall('([0-9]+)(?:-([0-9]+))?', _missing):
+            if end:
+                missing.extend(range(int(start), int(end)+1))
+            else:
+                missing.append(int(start))
+        files[os.path.basename(name)] = (statements, missed, covered, missing)
+    assert  5 not in files[module_path][-1], files[module_path]
+    assert  6 not in files[module_path][-1], files[module_path]
+    assert  7 not in files[module_path][-1], files[module_path]
+    assert 11 not in files[module_path][-1], files[module_path]
+
+
+def run_xml_report():
+    stdout = run_coverage_command('xml', '-o', '-')
+    print(stdout)
+
+    import xml.etree.ElementTree as etree
+    data = etree.fromstring(stdout)
+
+    files = {}
+    for module in data.iterfind('.//class'):
+        files[module.get('filename').replace('\\', '/')] = dict(
+            (int(line.get('number')), int(line.get('hits')))
+            for line in module.findall('lines/line')
+        )
+
+    module_path = 'src/pkg/module1.pyx'
+
+    assert files[module_path][5] > 0, files[module_path]
+    assert files[module_path][6] > 0, files[module_path]
+    assert files[module_path][7] > 0, files[module_path]
+    assert files[module_path][11] > 0, files[module_path]
+
+
+def run_html_report():
+    from collections import defaultdict
+
+    stdout = run_coverage_command('html', '-d', 'html')
+    # coverage 6.1+ changed the order of the attributes => need to parse them separately
+    _parse_id = re.compile(r'id=["\'][^0-9"\']*(?P<id>[0-9]+)[^0-9"\']*["\']').search
+    _parse_state = re.compile(r'class=["\'][^"\']*(?P<state>mis|run|exc)[^"\']*["\']').search
+
+    files = {}
+    for file_path in iglob('html/*.html'):
+        with open(file_path) as f:
+            page = f.read()
+        report = defaultdict(set)
+        for line in re.split(r'id=["\']source["\']', page)[-1].splitlines():
+            lineno = _parse_id(line)
+            state = _parse_state(line)
+            if not lineno or not state:
+                continue
+            report[state.group('state')].add(int(lineno.group('id')))
+        files[file_path] = report
+
+    file_report = [data for path, data in files.items() if 'module1' in path][0]
+    executed, missing = file_report["run"], file_report["mis"]
+    assert executed
+    assert 5 in executed, executed
+    assert 6 in executed, executed
+    assert 7 in executed, executed
+    assert 11 in executed, executed
+
+
+if __name__ == '__main__':
+    run_report()
+    run_xml_report()
+    run_html_report()