Repository: HypothesisWorks/hypothesis

Issue:
Log message hiding interfers with pytest log caputure fixture.
The log message hiding merged in #1535 interferes with pytests log capturing. 

In the example below I am using the [caplog fixture](https://docs.pytest.org/en/latest/logging.html#caplog-fixture) to capture the log message and test that they are correct.
In this simple case just asserting the number of messages. This results in an empty record list in the first call with  3.69.11 but works as expected with 3.69.10

```python
import logging
from hypothesis import given, strategies

@given(x=strategies.floats(min_value=-3, max_value=3))
def test_logcapture(x, caplog):

    with caplog.at_level(logging.CRITICAL, logger='root'):
        caplog.clear()
        logging.critical(f"x is {x}")
    assert len(caplog.records) == 1 
```


Discussion:
...Yep, I can see how this is doing the wrong thing, but I don't know what the right behaviour might be.  @jenshnielsen @DRMacIver any ideas?  The goal is that 

1. Each test case has the normal logging config, whatever that is for the user
2. When a test fails, only the logging from the last test case is observable

Unfortunately #1535 gave up (1) to achieve (2).  Should we just revert it and go back to the drawing board?
I don't know much about how logging works internally, but is it possible to in some way mess with it so that logging is enabled (ed: This originally said "disabled", which caused this sentence to not make any sense) but the actual printing to screen is turned off? e.g. it's a bit aggressive, but we could actually try messing with the raw file handles and redirect them inside non-final tests. Anything that relies on output capturing would still work, but anything that actually prints to the screen would not.

(Making that work across unixish OSes and Windows will be a *pain*).

My reluctant conclusion is that reverting #1535 is probably going to be the right thing to do regardless of what the correct solution is though. 😢 Logging shouldn't be disabled if we want this sort of test to work, regardless of what the actual solution we implement is.
The best would be if this could grab the handler that is responsible for printing to the screen / std_out and disable the output for that one only. I find  https://docs.python.org/2/_images/logging_flow.png useful for understanding how the logger logic works
Disabling a streamhandler would break tests that capture and check stdout and stderr though :sob: 

I think we have to take this out of the Hypothesis core entirely, though we may be able to do something in the pytest plugin to clear the captured logs at the start of every test case.  This would achieve (1) and *partly* (2), by assuming that logging is only observable via the pytest log capture mechanism.

Resolving patch:
diff --git a/hypothesis-python/src/hypothesis/control.py b/hypothesis-python/src/hypothesis/control.py
--- a/hypothesis-python/src/hypothesis/control.py
+++ b/hypothesis-python/src/hypothesis/control.py
@@ -17,7 +17,6 @@
 
 from __future__ import division, print_function, absolute_import
 
-import logging
 import traceback
 
 from hypothesis import Verbosity, settings
@@ -67,20 +66,14 @@ def __init__(self, data, is_final=False, close_on_capture=True):
         self.close_on_capture = close_on_capture
         self.close_on_del = False
         self.notes = []
-        self.original_logging_disable = logging.NOTSET
 
     def __enter__(self):
-        if not self.is_final:
-            self.original_logging_disable = logging.root.manager.disable
-            logging.disable(logging.CRITICAL)
         self.assign_variable = _current_build_context.with_value(self)
         self.assign_variable.__enter__()
         return self
 
     def __exit__(self, exc_type, exc_value, tb):
         self.assign_variable.__exit__(exc_type, exc_value, tb)
-        if not self.is_final:
-            logging.disable(self.original_logging_disable)
         if self.close() and exc_type is None:
             raise CleanupFailed()
 


Test patch:
diff --git a/hypothesis-python/tests/cover/test_logging.py b/hypothesis-python/tests/cover/test_logging.py
deleted file mode 100644
--- a/hypothesis-python/tests/cover/test_logging.py
+++ /dev/null
@@ -1,62 +0,0 @@
-# coding=utf-8
-#
-# This file is part of Hypothesis, which may be found at
-# https://github.com/HypothesisWorks/hypothesis-python
-#
-# Most of this work is copyright (C) 2013-2018 David R. MacIver
-# (david@drmaciver.com), but it contains contributions by others. See
-# CONTRIBUTING.rst for a full list of people who may hold copyright, and
-# consult the git log if you need to determine who owns an individual
-# contribution.
-#
-# This Source Code Form is subject to the terms of the Mozilla Public License,
-# v. 2.0. If a copy of the MPL was not distributed with this file, You can
-# obtain one at http://mozilla.org/MPL/2.0/.
-#
-# END HEADER
-
-from __future__ import division, print_function, absolute_import
-
-import logging
-
-import pytest
-
-from hypothesis import given, reporting
-from hypothesis.strategies import integers
-
-
-def test_log_messages_are_dropped_when_executing_each_nonfinal_case(caplog):
-    # Setup a custom logger, and set the global minimum log level
-    logger = logging.getLogger('hypothesis.test.test_logging.example')
-    logging.disable(logging.INFO)
-
-    # Setup a trivial test function
-    @given(integers())
-    def test(x):
-        # This log message should always be dropped because of the global
-        # `logging.disable()` setting
-        logger.info('log info %d', x)
-
-        # This log message should only appear in the final test case
-        logger.warning('log warning %d', x)
-
-        # The test case needs to fail otherwise there won't be a final test
-        # case. Additionally, we want it to fail with a non-trivial minimal
-        # example
-        assert x < 10
-
-    # Exercise
-    with pytest.raises(AssertionError):
-        with reporting.with_reporter(reporting.default):
-            test()
-
-    # Verify that only one log record was produced, even though we executed
-    # multiple test cases
-    assert len(caplog.records) == 1, \
-        'Log messages from interim test executions should be suppressed'
-    record = caplog.records[0]
-    assert record.levelno == logging.WARNING, \
-        'The log message at INFO level should be suppressed by the pre-set ' \
-        'logging.disable() call'
-    assert record.message == 'log warning 10', \
-        'The log message should refer to the final test execution'