Repository: cython/cython

Issue:
Cython.Coverage: error with pyx-file coverage with lower-case filenames and wrong line numbers in .coverage
On windows, 
with cython0.27
and 
Coverage.py, version 4.3.4 with C extension

I try to report the coverage of a pyx-file with the Cython.Coverage plugin for coverage.py
`coverage run --source=src -m py.test --pyargs cythonarrays` the .coverage-file including line-numbers that were hit for the .pyx files.
However, `coverage report` issues 

> src\cythonarrays\array_shapes.pyx   TypeError: 'NoneType' object is not iterable
> src\cythonarrays\tests\example_cython.pyx   TypeError: 'NoneType' object is not iterable

Unfortunately, the coverage module catches the exception and did not show where the error occured.
So i had to patch coverage.summary.py to get the tracebak, which showed me, that set(self._code), becuase self._code was None in line 286 in 

```
class CythonModuleReporter(FileReporter):
    def lines(self):
        return set(self._code)
```

Further debugging led me to the point, that the "None" of self._code came from

```
class Plugin(CoveragePlugin):
    def _parse_lines(self, c_file, sourcefile):
        if sourcefile not in self._c_files_map:
            return (None,) * 2  # e.g. shared library file
```

and sourcefile was not in self._c_files_map, because sourcefile started with C:\Dev\cythonarrays\..., while the key in the self._c_files_map-dictionary was written with lowercase c:\dev\cythonarrays...
So abspath returns the path with lowercase because it finds the files in PATH. I installed the package ediable with pip install -e cythonarrays and this lead to entries in easyinstall.pth with lowercase path.

To patch the problem, i converted all filepaths to lowercase

```
class Plugin(CoveragePlugin):

    def file_reporter(self, filename)
        filename = os.path.abspath(filename).lower()

    def _parse_lines(self, c_file, sourcefile):
        (...)
        for filename, code in code_lines.items():
            abs_path = _find_dep_file_path(c_file, filename).lower()
```
This adds the *.pyx-files to the report.

However, this might be not the best solution, which works always on windows and linux...
Maybe you have another idea how to fix this.

The other problem are the line numbers reported by `coverage run` and stored in the .coverage result file, for the .pyx-file, which do not match the actual line numbers in the pyx-file.
In the .coverage -file, for my example_cython.pyx-file there are 26 line numbers between 1 and 983 collected.
`!coverage.py: This is a private format, don't read it directly!{"lines":{"C:\\Dev\\cythonarrays\\code\\cythonarrays\\src\\cythonarrays\\tests\\example_cython.pxd":[],"C:\\Dev\\cythonarrays\\code\\cythonarrays\\src\\cythonarrays\\tests\\example_cython.pyx":[1,246,10,288,207,983,537,155,284,285,286,31,32,289,290,291,36,37,287,243,244,245,54,55,314,315]},"file_tracers":{"C:\\Dev\\cythonarrays\\code\\cythonarrays\\src\\cythonarrays\\tests\\example_cython.pyx":"Cython.Coverage.Plugin","C:\\Dev\\cythonarrays\\code\\cythonarrays\\src\\cythonarrays\\tests\\example_cython.pxd":"Cython.Coverage.Plugin","C:\\Dev\\cythonarrays\\code\\cythonarrays\\src\\cythonarrays\\array_shapes.pyx":"Cython.Coverage.Plugin"}}`

 The  example_cython.pyx file has only 86 lines. 

The C-File i generated contains comments for the traced lines in the actual .pyx-file like:

>  /* "cythonarrays/tests/example_cython.pyx":31
 
to C-imported files like:
 

> /* "../../../../Anaconda/envs/cythonarrays/lib/site-packages/Cython/Includes/numpy/__init__.pxd":220

and to View.MemoryView and to a "Tree fragment"

> /* "View.MemoryView":120
>  /* "(tree fragment)":1

So the strange numbers the Cython.Coverage plugin collected in the c-file are mostly line numbers refering to the "View.MemoryView" 

```
  /* "View.MemoryView":983
 *         return self.from_object
 * 
 *     __pyx_getbuffer = capsule(<void *> &__pyx_memoryview_getbuffer, "getbuffer(obj, view, flags)")             # <<<<<<<<<<<<<<
 * 
 * 
 */
```

or to the "tree fragment"

 ```
 /* "(tree fragment)":1
 * def __reduce_cython__(self):             # <<<<<<<<<<<<<<
 *     cdef bint use_setstate
 *     state = (self.INF_d, self.INF_f, self.NAN_d, self.NAN_f, self.NINF_d, self.NINF_f, self._invalid_g, self._jobs_j, self._km_ij, self._not_initialized_ij, self._param_g, self._persons_gi, self._trips_ij, self._valid_g, self._zonenumbers_i, self.destinations, self.groups, self.n_threads, self.origins)
 */
```
What could be the reason that these line numbers are collected instead of the numbers of the lines in the actual .pyx-file, that were hit?




Discussion:
I found an alternative to .lower(), that helped to get the right line_numbers:

Instead of converting the filepath to lowercase, the following function returnes the filepath as the file system returnes it (on Windows - on linux it remains as it is):

```
import glob
def get_actual_filename(name):
    if not sys.platform.startswith('win'):
        return name
    dirs = name.split('\\')
    # disk letter
    test_name = [dirs[0].upper()]
    for d in dirs[1:]:
        test_name += ["%s[%s]" % (d[:-1], d[-1])]
    res = glob.glob('\\'.join(test_name))
    if not res:
        #File not found
        return None
    return res[0]
```
(thanks to [this link](https://stackoverflow.com/questions/3692261/in-python-how-can-i-get-the-correctly-cased-path-for-a-file)

Then the filename then can be converted with 
```
class Plugin(CoveragePlugin):

    def file_reporter(self, filename)
        filename = get_actual_filename(os.path.abspath(filename))

    def _parse_lines(self, c_file, sourcefile):
        (...)
        for filename, code in code_lines.items():
            abs_path = get_actual_filename(_find_dep_file_path(c_file, filename))
```



Resolving patch:
diff --git a/Cython/Coverage.py b/Cython/Coverage.py
--- a/Cython/Coverage.py
+++ b/Cython/Coverage.py
@@ -12,6 +12,7 @@
 from collections import defaultdict
 
 from coverage.plugin import CoveragePlugin, FileTracer, FileReporter  # requires coverage.py 4.0+
+from coverage.files import canonical_filename
 
 from .Utils import find_root_package_dir, is_package_dir, open_source_file
 
@@ -41,8 +42,8 @@ def _find_dep_file_path(main_file, file_path):
         for sys_path in sys.path:
             test_path = os.path.realpath(os.path.join(sys_path, file_path))
             if os.path.exists(test_path):
-                return test_path
-    return abs_path
+                return canonical_filename(test_path)
+    return canonical_filename(abs_path)
 
 
 class Plugin(CoveragePlugin):
@@ -63,7 +64,7 @@ def file_tracer(self, filename):
         if filename.startswith('<') or filename.startswith('memory:'):
             return None
         c_file = py_file = None
-        filename = os.path.abspath(filename)
+        filename = canonical_filename(os.path.abspath(filename))
         if self._c_files_map and filename in self._c_files_map:
             c_file = self._c_files_map[filename][0]
 
@@ -91,7 +92,7 @@ def file_reporter(self, filename):
         #    from coverage.python import PythonFileReporter
         #    return PythonFileReporter(filename)
 
-        filename = os.path.abspath(filename)
+        filename = canonical_filename(os.path.abspath(filename))
         if self._c_files_map and filename in self._c_files_map:
             c_file, rel_file_path, code = self._c_files_map[filename]
         else:


Test patch:
diff --git a/tests/run/coverage_api.srctree b/tests/run/coverage_api.srctree
--- a/tests/run/coverage_api.srctree
+++ b/tests/run/coverage_api.srctree
@@ -2,7 +2,7 @@
 # tag: coverage,trace
 
 """
-PYTHON -c 'import shutil; shutil.copy("pkg/coverage_test_pyx.pyx", "pkg/coverage_test_pyx.pxi")'
+PYTHON -c "import shutil; shutil.copy('pkg/coverage_test_pyx.pyx', 'pkg/coverage_test_pyx.pxi')"
 PYTHON setup.py build_ext -i
 PYTHON coverage_test.py
 """
@@ -14,7 +14,8 @@ from Cython.Build import cythonize
 
 setup(ext_modules = cythonize([
     'coverage_test_*.py*',
-    'pkg/coverage_test_*.py*'
+    'pkg/coverage_test_*.py*',
+    'Package2/CoverageTest_*.py*'
 ]))
 
 
@@ -68,6 +69,52 @@ def main_func(int x):                            # 11
     return cfunc1(x) + func1(x, 4) + func2(x)    # 12
 
 
+######## Package2/__init__.py ########
+# Add MixedCase package and filenames to test if the files are found
+
+######## Package2/CoverageTest_py.py ########
+# cython: linetrace=True
+# distutils: define_macros=CYTHON_TRACE=1
+
+def func1(a, b):
+    x = 1               #  5
+    c = func2(a) + b    #  6
+    return x + c        #  7
+
+
+def func2(a):
+    return a * 2        # 11
+
+
+######## Package2/CoverageTest_pyx.pyx ########
+# cython: linetrace=True
+# distutils: define_macros=CYTHON_TRACE=1
+
+def func1(int a, int b):
+    cdef int x = 1      #  5
+    c = func2(a) + b    #  6
+    return x + c        #  7
+
+
+def func2(int a):
+    return a * 2        # 11
+
+
+######## coverage_test_include_pyx.pyx ########
+# cython: linetrace=True
+# distutils: define_macros=CYTHON_TRACE=1
+
+cdef int x = 5                                   #  4
+
+cdef int cfunc1(int x):                          #  6
+    return x * 3                                 #  7
+
+include "pkg/coverage_test_pyx.pxi"              #  9
+
+def main_func(int x):                            # 11
+    return cfunc1(x) + func1(x, 4) + func2(x)    # 12
+
+
 ######## coverage_test.py ########
 
 import re
@@ -84,8 +131,12 @@ from pkg import coverage_test_py
 from pkg import coverage_test_pyx
 import coverage_test_include_pyx
 
+# test the MixedCase Files and packages
+from Package2 import CoverageTest_py
+from Package2 import CoverageTest_pyx
 
-for module in [coverage_test_py, coverage_test_pyx, coverage_test_include_pyx]:
+for module in [coverage_test_py, coverage_test_pyx, coverage_test_include_pyx,
+               CoverageTest_py, CoverageTest_pyx]:
     assert not any(module.__file__.endswith(ext) for ext in '.py .pyc .pyo .pyw .pyx .pxi'.split()), \
         module.__file__
 
@@ -93,10 +144,18 @@ for module in [coverage_test_py, coverage_test_pyx, coverage_test_include_pyx]:
 def source_file_for(module):
     module_name = module.__name__
     path, ext = os.path.splitext(module.__file__)
-    platform_suffix = re.search(r'[.](?:cpython|pypy)-[0-9]+[^.]*$', path, re.I)
-    if platform_suffix:
-        path = path[:platform_suffix.start()]
-    return path + '.' + module_name.rsplit('_', 1)[-1]
+    if ext == '.so':
+        # Linux/Unix/Mac extension module
+        platform_suffix = re.search(r'[.](?:cpython|pypy)-[0-9]+[-_a-z0-9]*$', path, re.I)
+        if platform_suffix:
+            path = path[:platform_suffix.start()]
+    elif ext == '.pyd':
+        # Windows extension module
+        platform_suffix = re.search(r'[.]cp[0-9]+-win[_a-z0-9]*$', path, re.I)
+        if platform_suffix:
+            path = path[:platform_suffix.start()]
+    source_filepath = path + '.' + module_name.rsplit('_', 1)[-1]
+    return source_filepath
 
 
 def run_coverage(module):
@@ -137,3 +196,5 @@ if __name__ == '__main__':
     run_coverage(coverage_test_py)
     run_coverage(coverage_test_pyx)
     run_coverage(coverage_test_include_pyx)
+    run_coverage(CoverageTest_py)
+    run_coverage(CoverageTest_pyx)
diff --git a/tests/run/coverage_cmd.srctree b/tests/run/coverage_cmd.srctree
--- a/tests/run/coverage_cmd.srctree
+++ b/tests/run/coverage_cmd.srctree
@@ -2,7 +2,7 @@
 # tag: coverage,trace
 
 """
-PYTHON -c 'import shutil; shutil.copy("pkg/coverage_test_pyx.pyx", "pkg/coverage_test_pyx.pxi")'
+PYTHON -c "import shutil; shutil.copy('pkg/coverage_test_pyx.pyx', 'pkg/coverage_test_pyx.pxi')"
 PYTHON setup.py build_ext -i
 PYTHON -m coverage run coverage_test.py
 PYTHON collect_coverage.py