Repository: HypothesisWorks/hypothesis

Issue:
Hypothesis>=3.31 breaks coverage tracking for test methods.
I was adding some new code to Twisted Klein and the code is tested with the help of hypothesis (thanks!):

https://github.com/twisted/klein/pull/220

The build is failing because of a drop in coverage, and the [coverage report](https://codecov.io/gh/twisted/klein/compare/61c06611324cde5cfb08be0f496a46b106fe2c40...f9bc624548b0bb5121859eb581be85ab867ae17a/diff#D3-61) shows that the code in the unit tests themselves has not been used (but the code being tested is covered) in tests where the `@given` decorator is used.

I'm pretty sure this is a new thing, so I tried older version and have determined that hypothesis 3.30 works, and 3.31 and later do not.



Discussion:
There were five versions of 3.30.x – from 3.30.0 to 3.30.4.

3.30.4 added a bunch of stuff for using coverage information, and seems like an obvious candidate for the bug.

3.31.0 blocked installation on Python 3.3, and I think is a red herring here.

Which patch release of 3.30 do you know works?
Interesting… I had assumed that Tox (pip?) would install `3.30.4` when I specified `3.30`, but I went ahead and tested specific minor versions and `3.30.0` works as expected, but `3.30.1` does not.

So it appears that `3.30.1` introduced the issue.

I actually think this is correct behaviour and suspect that the thing that is being diffed against is a Hypothesis run from when the behaviour was broken. 3.30.1 fixed the problem that when run using Hypothesis 3.30.0 you'd get a whole bunch of files you didn't ask for in your coverage. Your [coveragerc](https://github.com/twisted/klein/blob/master/.coveragerc) is configured to only include things under klein, so the test files *shouldn't* be included.
>  Your coveragerc is configured to only include things under klein, so the test files shouldn't be included.

Oh, wait, that's not true - you've got your tests under the klein namespace. Hmm.
> I actually think this is correct behaviour and suspect that the thing that is being diffed against…

There is no thing being differ against.  I'm running coverage on my computer and when I run my tests with hypothesis `3.30.1` or greater the coverage report shows that tests decorated with `@given` have not been run.  If I use hypothesis `3.30.0`, then show as covered.  I'm not using any diff thingo (I don't even know how to diff coverage reports, actually).
Yeah, but my point was that the fact that tests show up in 3.30.0 could just be the fact that 3.30.0 is wrong about what files should be included in coverage! Though having re-examined coverage's behaviour I don't think that's it.
I have the same issue. I tried to come up with an simple example, but I failed. The only thing I can say: the coverage is correct if I set use_coverage=False in conftest.py, otherwise it fails consistently.
We are guilty of having the tests inside our project's namespace and therefore running coverage on them. It is nice to see that all the code of the tests runs, though. And it is test code that is not tracked.
OK. I understand this bug now I think. At the very least I know the culprit and a workaround, though I've not quite nailed down the interactions that cause it, and I'm not sure what the best fix is yet.

TLDR workaround: Add ` */test_*.py` to the source section of your coveragerc.

The culprit is the fact that we lie about what file `@given` tests come from, using [this lovely little function](https://github.com/HypothesisWorks/hypothesis-python/blob/master/src/hypothesis/internal/compat.py#L361). This is largely for aesthetic reasons, but the aesthetic reasons are quite a big deal.

What I *think* is then happening is that coverage is tracing the call there, looking up `__file__` in the function's globals, using that to conclude it's not a test you want to run, and then caching the result so that when it calls your actual code you don't recheck with the new frame.

There are some subtleties that I haven't quite got though. In particular I haven't found a good reproducible test case yet, and some things I think shouldn't work do work.
I've been experimenting more with this today, and the really annoying thing is that I simply cannot reproduce this in an isolated way. It happens in klein, but whenever I try to produce a test case that demonstrates it it simply doesn't work. I don't currently have any idea why this might be.

I'm more than happy to remove the functionality that is breaking this and collaborate with the pytest folks about a less terrible way to do it, but I don't really want to do a fix for this without a test to verify that it stays fixed.

If anyone has any ideas it would be much appreciated.
Here's a somewhat minimal example that reproduces the behavior for me, derived from the txkube test suite (perhaps obviously):

https://github.com/LeastAuthority/txkube/blob/53ddb656b222b7926b06e6f396b3dada982c2215/src/txkube/test/test_authentication.py

When I run this test and generate a coverage report:

    coverage run --rcfile=${PWD}/.coveragerc $(type -p trial) txkube.test.test_authentication
    coverage report

Then I am told there are uncovered lines in that test:

```
Name                                       Stmts   Miss Branch BrPart  Cover
----------------------------------------------------------------------------
src/txkube/test/test_authentication.py         6      1      0      0    83%
```
But in my estimation I would say it is trivially obvious that all lines should be covered.

Hope this helps.

Resolving patch:
diff --git a/src/hypothesis/core.py b/src/hypothesis/core.py
--- a/src/hypothesis/core.py
+++ b/src/hypothesis/core.py
@@ -413,7 +413,22 @@ def __init__(
 
         if settings.use_coverage and not IN_COVERAGE_TESTS:  # pragma: no cover
             if Collector._collectors:
-                self.hijack_collector(Collector._collectors[-1])
+                parent = Collector._collectors[-1]
+
+                # We include any files the collector has already decided to
+                # trace whether or not on re-investigation we still think it
+                # wants to trace them. The reason for this is that in some
+                # cases coverage gets the wrong answer when we run it
+                # ourselves due to reasons that are our fault but are hard to
+                # fix (we lie about where certain functions come from).
+                # This causes us to not record the actual test bodies as
+                # covered. But if we intended to trace test bodies then the
+                # file must already have been traced when getting to this point
+                # and so will already be in the collector's data. Hence we can
+                # use that information to get the correct answer here.
+                # See issue 997 for more context.
+                self.files_to_propagate = set(parent.data)
+                self.hijack_collector(parent)
 
             self.collector = Collector(
                 branch=True,
@@ -563,14 +578,14 @@ def save_data(covdata):
                 covdata.add_arcs({
                     filename: {
                         arc: None
-                        for arc in self.coverage_data.arcs(filename)}
+                        for arc in self.coverage_data.arcs(filename) or ()}
                     for filename in self.files_to_propagate
                 })
             else:
                 covdata.add_lines({
                     filename: {
                         line: None
-                        for line in self.coverage_data.lines(filename)}
+                        for line in self.coverage_data.lines(filename) or ()}
                     for filename in self.files_to_propagate
                 })
             collector.save_data = original_save_data


Test patch: