Repository: HypothesisWorks/hypothesis

Issue:
Add compat with pytest-capturelog for note()-style logging capture?
Using hypothesis with code that has logging statements makes it very difficult to use those log calls to debug.

Having the ability to have logging output captured in the style of `note()` would be extremely useful. The [pytest-capturelog](https://pypi.python.org/pypi/pytest-capturelog) plugin collects the logging output into the test failure message. It would be really nice to have some kind of cross-compatibility with them so that it can group captured logs by example rather than at the test-function level



Discussion:
As of pytest 3.3, log capturing is built in - meaning that logs are probably being captured (for all examples, eek) more often, *and* there's a standard thing that we can be compatible with in the pytest plugin (in hypothesis.extra)

See https://docs.pytest.org/en/latest/logging.html
This may be as simple as disabling logging for all but the final example of a test, which can be done in `hypothesis.control.BuildContext`:

```python
if not self.is_final:  # in __enter__
    logging.disable()
...
if not self.is_final:  # in __exit__
    logging.disable(logging.NOTSET)
```

and adding a test in `tests/pytest/` that a test with a log statement only logs it once.

Resolving patch:
diff --git a/hypothesis-python/src/hypothesis/control.py b/hypothesis-python/src/hypothesis/control.py
--- a/hypothesis-python/src/hypothesis/control.py
+++ b/hypothesis-python/src/hypothesis/control.py
@@ -17,6 +17,7 @@
 
 from __future__ import division, print_function, absolute_import
 
+import logging
 import traceback
 
 from hypothesis import Verbosity, settings
@@ -66,14 +67,20 @@ def __init__(self, data, is_final=False, close_on_capture=True):
         self.close_on_capture = close_on_capture
         self.close_on_del = False
         self.notes = []
+        self.original_logging_disable = logging.NOTSET
 
     def __enter__(self):
+        if not self.is_final:
+            self.original_logging_disable = logging.root.manager.disable
+            logging.disable(logging.CRITICAL)
         self.assign_variable = _current_build_context.with_value(self)
         self.assign_variable.__enter__()
         return self
 
     def __exit__(self, exc_type, exc_value, tb):
         self.assign_variable.__exit__(exc_type, exc_value, tb)
+        if not self.is_final:
+            logging.disable(self.original_logging_disable)
         if self.close() and exc_type is None:
             raise CleanupFailed()
 


Test patch:
diff --git a/hypothesis-python/tests/cover/test_explicit_examples.py b/hypothesis-python/tests/cover/test_explicit_examples.py
--- a/hypothesis-python/tests/cover/test_explicit_examples.py
+++ b/hypothesis-python/tests/cover/test_explicit_examples.py
@@ -203,7 +203,7 @@ def test(x):
         with reporting.with_reporter(reporting.default):
             test()
     ls = out.getvalue().splitlines()
-    assert ls == [u"x -> 1", 'x -> 2', 'x -> 3']
+    assert ls == [u"x -> 1", u"x -> 2", u"x -> 3"]
 
 
 def test_prints_note_in_failing_example():
diff --git a/hypothesis-python/tests/cover/test_logging.py b/hypothesis-python/tests/cover/test_logging.py
new file mode 100644
--- /dev/null
+++ b/hypothesis-python/tests/cover/test_logging.py
@@ -0,0 +1,62 @@
+# coding=utf-8
+#
+# This file is part of Hypothesis, which may be found at
+# https://github.com/HypothesisWorks/hypothesis-python
+#
+# Most of this work is copyright (C) 2013-2018 David R. MacIver
+# (david@drmaciver.com), but it contains contributions by others. See
+# CONTRIBUTING.rst for a full list of people who may hold copyright, and
+# consult the git log if you need to determine who owns an individual
+# contribution.
+#
+# This Source Code Form is subject to the terms of the Mozilla Public License,
+# v. 2.0. If a copy of the MPL was not distributed with this file, You can
+# obtain one at http://mozilla.org/MPL/2.0/.
+#
+# END HEADER
+
+from __future__ import division, print_function, absolute_import
+
+import logging
+
+import pytest
+
+from hypothesis import given, reporting
+from hypothesis.strategies import integers
+
+
+def test_log_messages_are_dropped_when_executing_each_nonfinal_case(caplog):
+    # Setup a custom logger, and set the global minimum log level
+    logger = logging.getLogger('hypothesis.test.test_logging.example')
+    logging.disable(logging.INFO)
+
+    # Setup a trivial test function
+    @given(integers())
+    def test(x):
+        # This log message should always be dropped because of the global
+        # `logging.disable()` setting
+        logger.info('log info %d', x)
+
+        # This log message should only appear in the final test case
+        logger.warning('log warning %d', x)
+
+        # The test case needs to fail otherwise there won't be a final test
+        # case. Additionally, we want it to fail with a non-trivial minimal
+        # example
+        assert x < 10
+
+    # Exercise
+    with pytest.raises(AssertionError):
+        with reporting.with_reporter(reporting.default):
+            test()
+
+    # Verify that only one log record was produced, even though we executed
+    # multiple test cases
+    assert len(caplog.records) == 1, \
+        'Log messages from interim test executions should be suppressed'
+    record = caplog.records[0]
+    assert record.levelno == logging.WARNING, \
+        'The log message at INFO level should be suppressed by the pre-set ' \
+        'logging.disable() call'
+    assert record.message == 'log warning 10', \
+        'The log message should refer to the final test execution'