Repository: googleapis/google-cloud-python

Issue:
Allowing logging system tests to fail.
These hose our builds. (See my branch name.)

FWIW Someone should also deal with the eventual-consistency issue with Pub/Sub snapshot system tests.


Discussion:
This is on @lukesneeringer
Also worth noting that @jonparrott suggested I used `pytest.mark.xfail` but we've got some confusion / disagreement about how it works. As a test (for anyone interested) make `test_it.py` with

```python
import random
import pytest


@pytest.mark.xfail
def test_rand():
    assert random.random() < 0.5
```

and running it

```
$ py.test test_it.py
=============== test session starts ===============
platform linux2 -- Python 2.7.13, pytest-3.1.0, py-1.4.34, pluggy-0.4.0
rootdir: ${CWD}, inifile: tox.ini
plugins: cov-2.4.0
collected 1 items

test_it.py x

=============== 1 xfailed in 0.00 seconds ===============
$ echo ${?}
0
$ py.test test_it.py
=============== test session starts ===============
platform linux2 -- Python 2.7.13, pytest-3.1.0, py-1.4.34, pluggy-0.4.0
rootdir: ${CWD}, inifile: tox.ini
plugins: cov-2.4.0
collected 1 items

test_it.py X

=============== 1 xpassed in 0.00 seconds ===============
$ echo ${?}
0
```
~My understanding about `@pytest.mark.xfail` is that it will fail the test if the test _does not_ fail, which is not really what you want for a flaky test.~

Which is wrong! Today I learned.
@lukesneeringer That **was** my own understanding (and @jonparrott's as well). Please run the [above][1] / confirm I'm not insane.

[1]: https://github.com/GoogleCloudPlatform/google-cloud-python/pull/3691#issuecomment-318442077
Yeah, it looks like `@pytest.mark.xfail` is not the semantic equivalent of `@unittest.expectedFailure`.

I have mixed feelings about this. _Both_ of those concepts have value.
/cc @duggelz and @liyanhui1228, who probably care quite a lot about this and may be able to improve these tests.
@jonparrott Do we need to wait for their feedback to tighten the tourniquet?  They can always open new PRs which fix the logging tests and remove the "allow-them-to-fail" switch.
@tseaver I'm okay to merge this, I just wanted Doug and Angela to be aware that it's happening.
Thanks for letting us know! This looks good.
Also FYI (whoever is going to fix the b0rken tests): The issue is in clean-up, when some resources seem to 404 (on delete) for 10+ minutes. ISTM that the 404 is "real" and clean-up is doing the wrong thing.
Can I get the link to the CircleCI build which has the error message for the failing system test? I'll try to fix it.
#3237 has a very long history of this failure. If you want a recent failure, just visit https://circleci.com/gh/GoogleCloudPlatform/google-cloud-python/tree/master, about 5 out of every 6 builds (I just made up that ratio) fails because of logging.
Thanks for this information, I'll look into it.
So this didn't actually fix the build b0rken-ness because of the CircleCI timeout ([e.g.][1]). I think we should reduce the number of retries / add `-s` to the command so that there won't be 10+ minutes without any output.

[1]: https://circleci.com/gh/GoogleCloudPlatform/google-cloud-python/2661
Yeah the CircleCI will kill the test if there are 10+ minutes without any output and won't care the success code we set here. We can change the max retries to 9 and then the time will become 1 + 2 + ... + 256 = 511 seconds which is less than 600 seconds time out threshold. I have prepared a PR for it.

Also question for the clean up issue, is the loggers that should be deleted have already been deleted somewhere other than in the tearDown method and then causing 404?

Resolving patch:
diff --git a/logging/nox.py b/logging/nox.py
--- a/logging/nox.py
+++ b/logging/nox.py
@@ -74,9 +74,10 @@ def system_tests(session, python_version):
     session.run(
         'py.test',
         '-vvv',
+        '-s',
         'tests/system.py',
         *session.posargs,
-        success_codes=range(0, 100),
+        success_codes=range(0, 100)
     )
 
 


Test patch:
diff --git a/logging/tests/system.py b/logging/tests/system.py
--- a/logging/tests/system.py
+++ b/logging/tests/system.py
@@ -116,7 +116,7 @@ def setUp(self):
         self._handlers_cache = logging.getLogger().handlers[:]
 
     def tearDown(self):
-        retry = RetryErrors(NotFound, max_tries=10)
+        retry = RetryErrors(NotFound, max_tries=9)
         for doomed in self.to_delete:
             retry(doomed.delete)()
         logging.getLogger().handlers = self._handlers_cache[:]