Repository: cython/cython
Pull request: Update Coverage.py 
this commit fixes #1898
use filepath with the correct upper and lowercases

Problem statement:
Cython.Coverage: error with pyx-file coverage with lower-case filenames and wrong line numbers in .coverage
On windows, 
with cython0.27
and 
Coverage.py, version 4.3.4 with C extension

I try to report the coverage of a pyx-file with the Cython.Coverage plugin for coverage.py
`coverage run --source=src -m py.test --pyargs cythonarrays` the .coverage-file including line-numbers that were hit for the .pyx files.
However, `coverage report` issues 

> src\cythonarrays\array_shapes.pyx   TypeError: 'NoneType' object is not iterable
> src\cythonarrays\tests\example_cython.pyx   TypeError: 'NoneType' object is not iterable

Unfortunately, the coverage module catches the exception and did not show where the error occured.
So i had to patch coverage.summary.py to get the tracebak, which showed me, that set(self._code), becuase self._code was None in line 286 in 

```
class CythonModuleReporter(FileReporter):
    def lines(self):
        return set(self._code)
```

Further debugging led me to the point, that the "None" of self._code came from

```
class Plugin(CoveragePlugin):
    def _parse_lines(self, c_file, sourcefile):
        if sourcefile not in self._c_files_map:
            return (None,) * 2  # e.g. shared library file
```

and sourcefile was not in self._c_files_map, because sourcefile started with C:\Dev\cythonarrays\..., while the key in the self._c_files_map-dictionary was written with lowercase c:\dev\cythonarrays...
So abspath returns the path with lowercase because it finds the files in PATH. I installed the package ediable with pip install -e cythonarrays and this lead to entries in easyinstall.pth with lowercase path.

To patch the problem, i converted all filepaths to lowercase

```
class Plugin(CoveragePlugin):

    def file_reporter(self, filename)
        filename = os.path.abspath(filename).lower()

    def _parse_lines(self, c_file, sourcefile):
        (...)
        for filename, code in code_lines.items():
            abs_path = _find_dep_file_path(c_file, filename).lower()
```
This adds the *.pyx-files to the report.

However, this might be not the best solution, which works always on windows and linux...
Maybe you have another idea how to fix this.

The other problem are the line numbers reported by `coverage run` and stored in the .coverage result file, for the .pyx-file, which do not match the actual line numbers in the pyx-file.
In the .coverage -file, for my example_cython.pyx-file there are 26 line numbers between 1 and 983 collected.
`!coverage.py: This is a private format, don't read it directly!{"lines":{"C:\\Dev\\cythonarrays\\code\\cythonarrays\\src\\cythonarrays\\tests\\example_cython.pxd":[],"C:\\Dev\\cythonarrays\\code\\cythonarrays\\src\\cythonarrays\\tests\\example_cython.pyx":[1,246,10,288,207,983,537,155,284,285,286,31,32,289,290,291,36,37,287,243,244,245,54,55,314,315]},"file_tracers":{"C:\\Dev\\cythonarrays\\code\\cythonarrays\\src\\cythonarrays\\tests\\example_cython.pyx":"Cython.Coverage.Plugin","C:\\Dev\\cythonarrays\\code\\cythonarrays\\src\\cythonarrays\\tests\\example_cython.pxd":"Cython.Coverage.Plugin","C:\\Dev\\cythonarrays\\code\\cythonarrays\\src\\cythonarrays\\array_shapes.pyx":"Cython.Coverage.Plugin"}}`

 The  example_cython.pyx file has only 86 lines. 

The C-File i generated contains comments for the traced lines in the actual .pyx-file like:

>  /* "cythonarrays/tests/example_cython.pyx":31
 
to C-imported files like:
 

> /* "../../../../Anaconda/envs/cythonarrays/lib/site-packages/Cython/Includes/numpy/__init__.pxd":220

and to View.MemoryView and to a "Tree fragment"

> /* "View.MemoryView":120
>  /* "(tree fragment)":1

So the strange numbers the Cython.Coverage plugin collected in the c-file are mostly line numbers refering to the "View.MemoryView" 

```
  /* "View.MemoryView":983
 *         return self.from_object
 * 
 *     __pyx_getbuffer = capsule(<void *> &__pyx_memoryview_getbuffer, "getbuffer(obj, view, flags)")             # <<<<<<<<<<<<<<
 * 
 * 
 */
```

or to the "tree fragment"

 ```
 /* "(tree fragment)":1
 * def __reduce_cython__(self):             # <<<<<<<<<<<<<<
 *     cdef bint use_setstate
 *     state = (self.INF_d, self.INF_f, self.NAN_d, self.NAN_f, self.NINF_d, self.NINF_f, self._invalid_g, self._jobs_j, self._km_ij, self._not_initialized_ij, self._param_g, self._persons_gi, self._trips_ij, self._valid_g, self._zonenumbers_i, self.destinations, self.groups, self.n_threads, self.origins)
 */
```
What could be the reason that these line numbers are collected instead of the numbers of the lines in the actual .pyx-file, that were hit?

Hints from issue discussion:
I found an alternative to .lower(), that helped to get the right line_numbers:

Instead of converting the filepath to lowercase, the following function returnes the filepath as the file system returnes it (on Windows - on linux it remains as it is):

```
import glob
def get_actual_filename(name):
    if not sys.platform.startswith('win'):
        return name
    dirs = name.split('\\')
    # disk letter
    test_name = [dirs[0].upper()]
    for d in dirs[1:]:
        test_name += ["%s[%s]" % (d[:-1], d[-1])]
    res = glob.glob('\\'.join(test_name))
    if not res:
        #File not found
        return None
    return res[0]
```
(thanks to [this link](https://stackoverflow.com/questions/3692261/in-python-how-can-i-get-the-correctly-cased-path-for-a-file)

Then the filename then can be converted with 
```
class Plugin(CoveragePlugin):

    def file_reporter(self, filename)
        filename = get_actual_filename(os.path.abspath(filename))

    def _parse_lines(self, c_file, sourcefile):
        (...)
        for filename, code in code_lines.items():
            abs_path = get_actual_filename(_find_dep_file_path(c_file, filename))
```

Commit message: Update Coverage.py

use filepath with the correct upper and lowercases

Patch to review:
diff --git a/Cython/Coverage.py b/Cython/Coverage.py
index 133137c751e..1bdc2f5cc30 100644
--- a/Cython/Coverage.py
+++ b/Cython/Coverage.py
@@ -9,6 +9,7 @@
 import re
 import os.path
 import sys
+import glob
 from collections import defaultdict
 
 from coverage.plugin import CoveragePlugin, FileTracer, FileReporter  # requires coverage.py 4.0+
@@ -45,6 +46,35 @@ def _find_dep_file_path(main_file, file_path):
     return abs_path
 
 
+def _get_actual_filename(filepath):
+    """
+    on windows:
+    search the file `filepath' on the file system 
+    and return it with the correct upper/lowercases
+    on linux: return the filepath unmodified
+    
+    Parameters
+    ----------
+    filepath : str
+    
+    Returns
+    -------
+    filepath : str
+    """
+    if not sys.platform.startswith('win'):
+        return filepath
+    dirs = filepath.split('\\')
+    # disk letter
+    test_name = [dirs[0].upper()]
+    for d in dirs[1:]:
+        test_name += ["%s[%s]" % (d[:-1], d[-1])]
+    res = glob.glob('\\'.join(test_name))
+    if not res:
+        #File not found
+        return filepath
+    return res[0]
+
+
 class Plugin(CoveragePlugin):
     # map from traced file paths to absolute file paths
     _file_path_map = None
@@ -91,7 +121,7 @@ def file_reporter(self, filename):
         #    from coverage.python import PythonFileReporter
         #    return PythonFileReporter(filename)
 
-        filename = os.path.abspath(filename)
+        filename = _get_actual_filename(os.path.abspath(filename))
         if self._c_files_map and filename in self._c_files_map:
             c_file, rel_file_path, code = self._c_files_map[filename]
         else:
@@ -213,7 +243,7 @@ def _parse_lines(self, c_file, sourcefile):
             self._c_files_map = {}
 
         for filename, code in code_lines.items():
-            abs_path = _find_dep_file_path(c_file, filename)
+            abs_path = _get_actual_filename(_find_dep_file_path(c_file, filename))
             self._c_files_map[abs_path] = (c_file, filename, code)
 
         if sourcefile not in self._c_files_map:


Code review comments:
diff_hunk: @@ -45,6 +46,35 @@ def _find_dep_file_path(main_file, file_path):
     return abs_path
 
 
+def _get_actual_filename(filepath):
+    """
+    on windows:
+    search the file `filepath' on the file system 
+    and return it with the correct upper/lowercases
+    on linux: return the filepath unmodified
+    
+    Parameters
+    ----------
+    filepath : str
+    
+    Returns
+    -------
+    filepath : str
+    """
+    if not sys.platform.startswith('win'):
+        return filepath
+    dirs = filepath.split('\\')
+    # disk letter
+    test_name = [dirs[0].upper()]
+    for d in dirs[1:]:
+        test_name += ["%s[%s]" % (d[:-1], d[-1])]
path: Cython/Coverage.py
text: @user1:
How to append to a list? Easy! Create a new one and concatenate it to the existing one! :D
(I know, it's now your code...)

@author:
Thats whats happes if you copy&paste random code ;-)
I fixed this

diff_hunk: @@ -45,6 +46,35 @@ def _find_dep_file_path(main_file, file_path):
     return abs_path
 
 
+def _get_actual_filename(filepath):
path: Cython/Coverage.py
text: @user1:
I think this should just use `os.path.normcase()`.