CI: Running tests with xvfb for the clipboard to work
State: closed

- [X] closes #26887
- [ ] tests added / passed
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

With some luck this should make the clipboard tests work in azure. @jorisvandenbossche @jreback can you have a look once the CI finishes?

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: CI
description: Continuous Integration

Comments:
No idea why `xvfb` manages to start in some Linux builds, and fails in others. The only difference should be the conda packages installed, that I don't think it should affect `xvfb`.

And it also worries me, that in my fork pipelines (I set it up to test changes to the CI), things fail that don't fail here. I feel like I was dealing with windows 95... :\
In the build `Linux py37_locale` this seems to be working as expected for what I see. Clipboard tests are usually shown in the print skipped section, and here they are not. So looks like the problem is that there is something different in the `Linux py35_compat` build, that is making `xvfb` fail to start.
I see that `xvfb` actually starts the first time, and fails the second execution. I guess the problem may be that it doesn't have time to switch off before it's reloaded, will see what I can do.
Maybe try in a step before?

On Travis, they explicitly say in their docs it needs to be run in the "before_script" section (so the section where the tests are run)
I think the `sleep 1` will fix it, actually I did another run and worked correctly. Will play a bit more with it.

@jorisvandenbossche not an expert, but what I'm doing now is a bit different than what we've got in travis. In travis we just start the xvfb, and leaving it running. Here I use `xvfb-run`, which is like a Python context manager, it wraps a command with the server. If we use this, can't really move to another step, as I'm calling xvfb, and xvfb is calling the tests.
Confirmed that the problem is that `xvfb` fails to start because it's still running from the previous execution.

I can do a sleep for now if you want to get this in faster, but I think the right solution is the approach in #24106

I see that in travis the same solution doesn't seem to work, and clipboard tests are skipped. :\
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26932?src=pr&el=h1) Report
> Merging [#26932](https://codecov.io/gh/pandas-dev/pandas/pull/26932?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/d39a6de66803b2a13d6bbb8aa98c56e90f95526c?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/26932/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/26932?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #26932      +/-   ##
==========================================
- Coverage   91.87%   91.86%   -0.01%     
==========================================
  Files         180      180              
  Lines       50716    50743      +27     
==========================================
+ Hits        46593    46617      +24     
- Misses       4123     4126       +3
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.46% <ø> (ø)` | :arrow_up: |
| #single | `41.12% <ø> (-0.07%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/26932?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/26932/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `88.88% <0%> (-11.12%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/26932/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `96.89% <0%> (-0.12%)` | :arrow_down: |
| [pandas/core/strings.py](https://codecov.io/gh/pandas-dev/pandas/pull/26932/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc3RyaW5ncy5weQ==) | `98.93% <0%> (ø)` | :arrow_up: |
| [pandas/core/indexes/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/26932/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9iYXNlLnB5) | `96.71% <0%> (ø)` | :arrow_up: |
| [pandas/core/internals/managers.py](https://codecov.io/gh/pandas-dev/pandas/pull/26932/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW50ZXJuYWxzL21hbmFnZXJzLnB5) | `95.21% <0%> (ø)` | :arrow_up: |
| [pandas/core/generic.py](https://codecov.io/gh/pandas-dev/pandas/pull/26932/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ2VuZXJpYy5weQ==) | `94.2% <0%> (ø)` | :arrow_up: |
| [pandas/core/series.py](https://codecov.io/gh/pandas-dev/pandas/pull/26932/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc2VyaWVzLnB5) | `93.66% <0%> (+0.01%)` | :arrow_up: |
| [pandas/core/indexes/multi.py](https://codecov.io/gh/pandas-dev/pandas/pull/26932/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9tdWx0aS5weQ==) | `95.73% <0%> (+0.06%)` | :arrow_up: |
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/pull/26932/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `90.94% <0%> (+0.1%)` | :arrow_up: |
| [pandas/io/formats/printing.py](https://codecov.io/gh/pandas-dev/pandas/pull/26932/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvcHJpbnRpbmcucHk=) | `86.72% <0%> (+1.16%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26932?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26932?src=pr&el=footer). Last update [d39a6de...004d2d0](https://codecov.io/gh/pandas-dev/pandas/pull/26932?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26932?src=pr&el=h1) Report
> Merging [#26932](https://codecov.io/gh/pandas-dev/pandas/pull/26932?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/d47947a79099d650b4bb981a38ce09881ec6f43d?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/26932/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/26932?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #26932      +/-   ##
==========================================
- Coverage   91.87%   91.86%   -0.01%     
==========================================
  Files         180      180              
  Lines       50743    50743              
==========================================
- Hits        46620    46616       -4     
- Misses       4123     4127       +4
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.46% <ø> (ø)` | :arrow_up: |
| #single | `41.1% <ø> (-0.09%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/26932?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/26932/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `88.88% <0%> (-11.12%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/26932/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `96.89% <0%> (-0.12%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26932?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26932?src=pr&el=footer). Last update [d47947a...3a6c641](https://codecov.io/gh/pandas-dev/pandas/pull/26932?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

> I see that in travis the same solution doesn't seem to work, and clipboard tests are skipped. :\

In the previous PR where I fixed those tests, I moved it back from run_tests.sh to before_script exactly because it was not working there (as said, it's in the documentation of travis it needs to be there). 

You're free to try to find a solution that works for both, but personally I wouldn't spend too much time on that if you have something that works for Azure.
Going for the simple hacking solution for now, so the tests start running.

If I have time I'll give another try to call pytest just once, that will solve the problem for azure, and simplify the CI significantly.
Closing this, as I think it make more sense to implement those changes in a fresh PR after #26949 is merged (which will simplify this significantly).