Repository: HypothesisWorks/hypothesis

Issue:
Hypothesis doesn't recognize unittest.SkipTest exception
# Narrative example of failure

I haven't had time to construct a minimal example, so I'll give a full example of failure. I've recently updated a test in my Django project which conditionally skips a test in cases which no longer make sense. 

The test in question is defined in an abstract base class of a couple other test classes which specialize it further. One branch of one test no longer makes sense in the context of one of the specialization classes. I can't solve this with `assume()` or `@unittest.skipIf()` because neither has enough contextual data. Instead, I `raise unittest.SkipTest()` at runtime if the test should be skipped.

The issue is that Hypothesis doesn't know about `unittest.SkipTest`, so produces failure output, as seen below:

```shell
(.venv) coriolinus$ ./manage.py test
Creating test database for alias 'default'...
Installed 39 object(s) from 1 fixture(s)
Installed 21 object(s) from 1 fixture(s)
Installed 9579 object(s) from 1 fixture(s)
Installed 5 object(s) from 1 fixture(s)
Installed 6 object(s) from 1 fixture(s)
.............................s.............................Falsifying example: test_admin_sees_everyone(self=<core.tests.test_user_profile_visibility.ProfileVisibilityTests testMethod=test_admin_sees_everyone>, data=data(...))
Draw 1: <User: alpha>
s..........................................
----------------------------------------------------------------------
Ran 102 tests in 58.985s

OK (skipped=2)
Destroying test database for alias 'default'...
(.venv) coriolinus$ pip list | grep hypothesis
hypothesis (3.6.1)
```

# Expected behavior

Hypothesis does not produce failure output if a test raises `unittest.SkipTest`.

# Observed behavior

Hypothesis produces failure output if a test raises `unittest.SkipTest`.
Suppress error reporting on unittest.SkipTest
Resolves #514. Test case per the example in that thread.

Not ready to merge – I’ll need to write some tests for Hypothesis CI, but that will need to wait until I get home this evening.


Discussion:
Thanks for the report! I had a quick go at trying to write an example – is this sort of like what you were doing?

```python
import unittest

from hypothesis import given
from hypothesis.strategies import integers


class DemoTest(unittest.TestCase):

    @given(integers())
    def test_to_be_skipped(self, x):
        if x == 0:
            raise unittest.SkipTest()
        else:
            assert x == 0


unittest.main()
```

```console
$ python issue514.py
Falsifying example: test_to_be_skipped(self=<__main__.DemoTest testMethod=test_to_be_skipped>, x=0)
s
----------------------------------------------------------------------
Ran 1 test in 0.021s

OK (skipped=1)
```

Sounds like we should just recognise `unittest.SkipTest` as a special exception – I get the same result whether I raise the exception directly or call `self.skipTest(reason)`. Not sure how hard that is off the top of my head.
Yes, that's the essence of the issue. Agreed with the proposed course of
action. I feel like I should have submitted a PR to go with this bug
report, but I don't know anything about Hypothesis' internals.

On Thu, Apr 13, 2017 at 3:29 PM, Alex Chan <notifications@github.com> wrote:

> Thanks for the report! I had a quick go at trying to write an example – is
> this sort of like what you were doing?
>
> import unittest
> from hypothesis import givenfrom hypothesis.strategies import integers
>
> class DemoTest(unittest.TestCase):
>
>     @given(integers())
>     def test_to_be_skipped(self, x):
>         if x == 0:
>             raise unittest.SkipTest()
>         else:
>             assert x == 0
>
>
> unittest.main()
>
> $ python issue514.pyFalsifying example: test_to_be_skipped(self=<__main__.DemoTest testMethod=test_to_be_skipped>, x=0)s----------------------------------------------------------------------Ran 1 test in 0.021s
> OK (skipped=1)
>
> Sounds like we should just recognise unittest.SkipTest as a special
> exception – I get the same result whether I raise the exception directly or
> call self.skipTest(reason). Not sure how hard that is off the top of my
> head.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/HypothesisWorks/hypothesis-python/issues/514#issuecomment-293895402>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AHdeTr5MlcE7IqXHHhCasQKBMK_wfBRDks5rviMxgaJpZM4M8sBy>
> .
>

So at a very quick glance, I think we need to make a change somewhere around [line&nbsp;464 of core.py](https://github.com/HypothesisWorks/hypothesis-python/blob/37b3a20083096318ed892384556b2fb33d40cbf8/src/hypothesis/core.py#L464) so that if we catch a `unittest.SkipTest` exception, we abort the entire test run.

I’m not sure what mechanisms we have for marking a test as skipped (as opposed to just silently dropping the error) – I’ll dig in a little further when I have more time.
> I’m not sure what mechanisms we have for marking a test as skipped (as opposed to just silently dropping the error)

I think just immediately reraising the error should have the desired effect shouldn't it?
> I think just immediately reraising the error should have the desired effect shouldn't it?

Oh yeah. PR incoming. :)
One thing to consider here is that there are a number of forks of unittest (e.g. unittest2). We might want to have an extensible mechanism of some sort to list test skipping exceptions.
pytest also has its own [skipping mechanism](https://docs.pytest.org/en/latest/skipping.html#imperative-xfail-from-within-a-test-or-setup-function) which presumably has the same problem (less noticeably because pytest won't output on a test failure, but we still probably shouldn't be running the test and shrink process here) and is another argument for an extensible list.
Added a prototype of a configurable list. It tries to infer some sensible defaults from the environment, at the cost of several module imports. Does this approach look sensible?
The approach looks fairly reasonable. I wonder if reraise\_exceptions would be a better name though? Or something like that. Nothing about what this does is actually about skipping except that that happens to be the example that motivated it.
Okay, I think the final version of the core patch is present. I’m trying to write some tests, but a little stymied by the fact that I’m having trouble reproducing the bug within the Hypothesis test suite.

* pytest never seemed to be affected, AFAICT
* I can’t get unittest to spit out the error message within my test
* I haven’t tried to test unittest2 or nose’s behaviour yet

And then extra weird things happen if you run a test with nose but it’s pytest that raises the skip exception… yik.
> I can’t get unittest to spit out the error message within my test

I think with all of these it'd be fairly reasonable to write tests that don't try to invoke the test function with the runner: Just call the function/method directly, assert that you caught the right exception, use capture\_out and assert that "Falsifying example" does not appear in the test output.

You'll need to switch to the default reporter as well so that pytest doesn't eat your output.

So basically something like https://github.com/HypothesisWorks/hypothesis-python/blob/master/tests/cover/test_arbitrary_data.py#L43
> pytest never seemed to be affected, AFAICT

Actually, here's a suggested test that's more independent of the manifestation and would have shown up under pytest: The reason pytest din't seem to be affected was that it was masking the problem.

Because the real problem is that Hypothesis is treating this as a failure and minimizing it rather than failing fast. So the test should also assert that the function under test is only called once if it raises a skip test immediately.
From #535:

> I’m adding reraise_exceptions in #515 as well.

So, thinking about it, I wonder if this actually makes sense in a setting. Can you think of a use case where this should actually be localized to specific tests? I feel like we might be better off just making it a giant global list with an API to add to it.
> Can you think of a use case where this should actually be localized to specific tests? I feel like we might be better off just making it a giant global list with an API to add to it.

No.

TBQH, I’m still a bit unsure as to when you’d ever use this _except_ for raising a "skip this test" exception.
> No.

In that case I propose that rather than it being a setting we have (internally) a big list of exceptions to ignore and a public API for adding to that list globally.

> TBQH, I’m still a bit unsure as to when you’d ever use this except for raising a "skip this test" exception.

Me too. 
> I propose that rather than it being a setting we have (internally) a big list of exceptions to ignore and a public API for adding to that list globally.

Agreed, but further: since “public API” = “documented”, and neither of us can think of a reason to do anything besides sensible defaults (at least, for now), I’d propose leaving this private until we have a good reason not to.
>  I’d propose leaving this private until we have a good reason not to.

👍 WFM.

Resolving patch:
diff --git a/src/hypothesis/core.py b/src/hypothesis/core.py
--- a/src/hypothesis/core.py
+++ b/src/hypothesis/core.py
@@ -438,6 +438,41 @@ def process_arguments_to_given(
     return arguments, kwargs, test_runner, search_strategy
 
 
+def skip_exceptions_to_reraise():
+    """Return a tuple of exceptions meaning 'skip this test', to re-raise.
+
+    This is intended to cover most common test runners; if you would
+    like another to be added please open an issue or pull request.
+
+    """
+    import unittest
+    # This is a set because nose may simply re-export unittest.SkipTest
+    exceptions = set([unittest.SkipTest])
+
+    try:  # pragma: no cover
+        from unittest2 import SkipTest
+        exceptions.add(SkipTest)
+    except ImportError:
+        pass
+
+    try:  # pragma: no cover
+        from pytest.runner import Skipped
+        exceptions.add(Skipped)
+    except ImportError:
+        pass
+
+    try:  # pragma: no cover
+        from nose import SkipTest as NoseSkipTest
+        exceptions.add(NoseSkipTest)
+    except ImportError:
+        pass
+
+    return tuple(sorted(exceptions, key=str))
+
+
+exceptions_to_reraise = skip_exceptions_to_reraise()
+
+
 class StateForActualGivenExecution(object):
 
     def __init__(self, test_runner, search_strategy, test, settings, random):
@@ -468,7 +503,7 @@ def evaluate_test_data(self, data):
         except (
             HypothesisDeprecationWarning, FailedHealthCheck,
             StopTest,
-        ):
+        ) + exceptions_to_reraise:
             raise
         except Exception:
             escalate_hypothesis_internal_error()
diff --git a/src/hypothesis/version.py b/src/hypothesis/version.py
--- a/src/hypothesis/version.py
+++ b/src/hypothesis/version.py
@@ -17,5 +17,5 @@
 
 from __future__ import division, print_function, absolute_import
 
-__version_info__ = (3, 13, 0)
+__version_info__ = (3, 13, 1)
 __version__ = '.'.join(map(str, __version_info__))


Test patch:
diff --git a/tests/cover/test_skipping.py b/tests/cover/test_skipping.py
new file mode 100644
--- /dev/null
+++ b/tests/cover/test_skipping.py
@@ -0,0 +1,49 @@
+# coding=utf-8
+#
+# This file is part of Hypothesis, which may be found at
+# https://github.com/HypothesisWorks/hypothesis-python
+#
+# Most of this work is copyright (C) 2013-2017 David R. MacIver
+# (david@drmaciver.com), but it contains contributions by others. See
+# CONTRIBUTING.rst for a full list of people who may hold copyright, and
+# consult the git log if you need to determine who owns an individual
+# contribution.
+#
+# This Source Code Form is subject to the terms of the Mozilla Public License,
+# v. 2.0. If a copy of the MPL was not distributed with this file, You can
+# obtain one at http://mozilla.org/MPL/2.0/.
+#
+# END HEADER
+
+from __future__ import division, print_function, absolute_import
+
+import unittest
+
+import pytest
+
+from hypothesis import given
+from hypothesis.core import exceptions_to_reraise
+from tests.common.utils import capture_out
+from hypothesis.strategies import integers
+
+
+@pytest.mark.parametrize('skip_exception', exceptions_to_reraise)
+def test_no_falsifying_example_if_unittest_skip(skip_exception):
+    """If a ``SkipTest`` exception is raised during a test, Hypothesis should
+    not continue running the test and shrink process, nor should it print
+    anything about falsifying examples."""
+
+    class DemoTest(unittest.TestCase):
+
+        @given(xs=integers())
+        def test_to_be_skipped(self, xs):
+            if xs == 0:
+                raise skip_exception
+            else:
+                assert xs == 0
+
+    with capture_out() as o:
+        suite = unittest.defaultTestLoader.loadTestsFromTestCase(DemoTest)
+        unittest.TextTestRunner().run(suite)
+
+    assert 'Falsifying example' not in o.getvalue()
diff --git a/tests/pytest/test_skipping.py b/tests/pytest/test_skipping.py
new file mode 100644
--- /dev/null
+++ b/tests/pytest/test_skipping.py
@@ -0,0 +1,45 @@
+# coding=utf-8
+#
+# This file is part of Hypothesis, which may be found at
+# https://github.com/HypothesisWorks/hypothesis-python
+#
+# Most of this work is copyright (C) 2013-2017 David R. MacIver
+# (david@drmaciver.com), but it contains contributions by others. See
+# CONTRIBUTING.rst for a full list of people who may hold copyright, and
+# consult the git log if you need to determine who owns an individual
+# contribution.
+#
+# This Source Code Form is subject to the terms of the Mozilla Public License,
+# v. 2.0. If a copy of the MPL was not distributed with this file, You can
+# obtain one at http://mozilla.org/MPL/2.0/.
+#
+# END HEADER
+
+from __future__ import division, print_function, absolute_import
+
+pytest_plugins = str('pytester')
+
+
+PYTEST_TESTSUITE = """
+from hypothesis import given
+from hypothesis.strategies import integers
+import pytest
+
+@given(xs=integers())
+def test_to_be_skipped(xs):
+    if xs == 0:
+        pytest.skip()
+    else:
+        assert xs == 0
+"""
+
+
+def test_no_falsifying_example_if_pytest_skip(testdir):
+    """If ``pytest.skip() is called during a test, Hypothesis should not
+    continue running the test and shrink process, nor should it print anything
+    about falsifying examples."""
+    script = testdir.makepyfile(PYTEST_TESTSUITE)
+    result = testdir.runpytest(script, '--verbose', '--strict', '-m',
+                               'hypothesis')
+    out = '\n'.join(result.stdout.lines)
+    assert 'Falsifying example' not in out