TST: clean up to change setup.cfg to `xfail_strict = True` (GH23057)
State: closed

- [x] closes #23057
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Hello @dcreekp! Thanks for updating the PR.

 - There are no PEP8 issues in the file [`pandas/tests/indexing/test_coercion.py`](https://github.com/dcreekp/pandas/blob/dac4c0d59eec46358d16f7f3d3c5dd54d10b58de/pandas/tests/indexing/test_coercion.py) !

 - There are no PEP8 issues in the file [`pandas/tests/io/test_parquet.py`](https://github.com/dcreekp/pandas/blob/dac4c0d59eec46358d16f7f3d3c5dd54d10b58de/pandas/tests/io/test_parquet.py) !

 - There are no PEP8 issues in the file [`pandas/tests/plotting/test_datetimelike.py`](https://github.com/dcreekp/pandas/blob/dac4c0d59eec46358d16f7f3d3c5dd54d10b58de/pandas/tests/plotting/test_datetimelike.py) !

 - There are no PEP8 issues in the file [`pandas/tests/plotting/test_frame.py`](https://github.com/dcreekp/pandas/blob/dac4c0d59eec46358d16f7f3d3c5dd54d10b58de/pandas/tests/plotting/test_frame.py) !

 - There are no PEP8 issues in the file [`pandas/tests/plotting/test_misc.py`](https://github.com/dcreekp/pandas/blob/dac4c0d59eec46358d16f7f3d3c5dd54d10b58de/pandas/tests/plotting/test_misc.py) !

 - There are no PEP8 issues in the file [`pandas/tests/plotting/test_series.py`](https://github.com/dcreekp/pandas/blob/dac4c0d59eec46358d16f7f3d3c5dd54d10b58de/pandas/tests/plotting/test_series.py) !

 - There are no PEP8 issues in the file [`pandas/tests/scalar/period/test_period.py`](https://github.com/dcreekp/pandas/blob/dac4c0d59eec46358d16f7f3d3c5dd54d10b58de/pandas/tests/scalar/period/test_period.py) !

 - There are no PEP8 issues in the file [`pandas/util/_test_decorators.py`](https://github.com/dcreekp/pandas/blob/dac4c0d59eec46358d16f7f3d3c5dd54d10b58de/pandas/util/_test_decorators.py) !



##### Comment last updated on November 15, 2018 at 20:09 Hours UTC
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/23721?src=pr&el=h1) Report
> Merging [#23721](https://codecov.io/gh/pandas-dev/pandas/pull/23721?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/b7bdf7cd24ed719e676c1eda50f486c43644ddca?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/23721/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/23721?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master   #23721   +/-   ##
=======================================
  Coverage   92.31%   92.31%           
=======================================
  Files         161      161           
  Lines       51562    51562           
=======================================
  Hits        47599    47599           
  Misses       3963     3963
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.71% <ø> (ø)` | :arrow_up: |
| #single | `42.44% <ø> (ø)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/23721?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/util/\_test\_decorators.py](https://codecov.io/gh/pandas-dev/pandas/pull/23721/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvX3Rlc3RfZGVjb3JhdG9ycy5weQ==) | `93.24% <ø> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/23721?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/23721?src=pr&el=footer). Last update [b7bdf7c...19bfbf3](https://codecov.io/gh/pandas-dev/pandas/pull/23721?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

Looks like there are some xfailing tests that are passing: https://travis-ci.org/pandas-dev/pandas/jobs/455669456#L2806

There may be others in https://travis-ci.org/pandas-dev/pandas/builds/455669455?utm_source=github_status&utm_medium=notification, I only looked at the first build.

The ones that are flaky should be `xfail(strict=False)`.
@TomAugspurger @jbrockmendel 
Oh I get it. 
At least for https://travis-ci.org/pandas-dev/pandas/jobs/455669458 and https://travis-ci.org/pandas-dev/pandas/jobs/455669460 they are xpassing when running py3.7 on travis, while running py3.6 locally they didn't xpass
this one https://travis-ci.org/pandas-dev/pandas/jobs/455669458
will probably still fail because it is the same as this https://github.com/pandas-dev/pandas/issues/23726
can you rebase
you are picking up other changes. merge master and push again.
better for this one
Diff looks good here. CI is still failing from tests xpassing. You may need to add conditions or `strict=False` to those xfails.

https://travis-ci.org/pandas-dev/pandas/jobs/461026497#L2987
can you merge master and let's see the ci run again.
lgtm. @TomAugspurger 
Thanks @dcreekp! This will be very useful.
Awesome.
Just to let you know, as python2.7 support is going to be dropped. There are two or three tests that are marked xfail only because of the test failing on 2.7.
@TomAugspurger @jreback