python-pillow/Pillow — pull request #760 by hugovk (2014-07-01 15:05:51+00:00)
Title: Show coverage and quality reports for each diff

About [`diff-cover`](https://github.com/edx/diff-cover):

> Diff coverage is the percentage of new or modified lines that are covered by tests. This provides a clear and achievable standard for code review: If you touch a line of code, that line should be covered. Code coverage is *every* developer's responsibility!

> The `diff-cover` command line tool compares an XML coverage report with the output of `git diff`. It then reports coverage information for lines in the diff.

So let's add the reports to the end of Travis. In addition to `diff-cover`, it comes with a `diff-quality` tool that can show pylint, pyflakes (my contribution) and pep8 violations. Pylint is a bit too verbose, but pyflakes (sometimes shows programming errors) and pep8 can be useful.

For illustration, here's an [example CI report](https://travis-ci.org/hugovk/Pillow/jobs/28871884) from a commit (in a throwaway branch) with some dummy changes:

```
-------------
Diff Coverage
Diff: origin/master...HEAD, staged, and unstaged changes
-------------
PIL/Image.py (75.0%): Missing line(s) 1070-1071
-------------
Total: 8 line(s)
Missing: 2 line(s)
Coverage: 75%
-------------
-------------
Diff Quality
Quality Report: pyflakes
Diff: origin/master...HEAD, staged, and unstaged changes
-------------
.travis.yml (100%)
PIL/Image.py (37.5%):
622: local variable 'a' is assigned to but never used
623: local variable 'b' is assigned to but never used
624: local variable 'c' is assigned to but never used
1070: local variable 'b' is assigned to but never used
1071: local variable 'c' is assigned to but never used
1071: undefined name 'undefined'
Scripts/diffcover-install.sh (100%)
Scripts/diffcover-run.sh (100%)
-------------
Total: 26 line(s)
Violations: 5 line(s)
% Quality: 80%
-------------
-------------
Diff Quality
Quality Report: pep8
Diff: origin/master...HEAD, staged, and unstaged changes
-------------
.travis.yml (100%)
PIL/Image.py (75.0%):
1070: E225 missing whitespace around operator
1071: E225 missing whitespace around operator
Scripts/diffcover-install.sh (100%)
Scripts/diffcover-run.sh (100%)
-------------
Total: 26 line(s)
Violations: 2 line(s)
% Quality: 92%
-------------
```

Note that I've only added it for Python 2.x builds. PyPy isn't included because it doesn't have coverage because it takes too long on PyPy. Python 3.x isn't included because it takes an extra 2 minutes (or 4m with optimisation) just to install diff-cover, mostly due to slow building of its dependencies (lxml, jinja2). I've reported it to diff-cover, and they may create Python wheels to speed it up. Installation for Python 2.x takes about 30 seconds without optimisation. It would take 1m30 with optimisation, but including `CFLAGS=-O0` speeds it up and I think 30 seconds is acceptable.