TST: Call tests just once with --dist=loadscope
State: closed

Another try to what was tried in #26949. Before this PR the tests are called twice, once in a core for the tests that affect shared results, and once in parallel for the rest.

This PR makes a single call, and tests in the same scope (class or module) are granted to run in the same core, so no shared data problems should happen.

The tests being very slow was possibly caused by the proxy env variables, and not `--dist=loadscope`. But please check how long tests took before merging.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: CI
description: Continuous Integration

Comments:
If this finally works and can be merged, next step is to reimplement #26932. It failed before in the second run of the tests, but after this is merged, we'll have only one run, and it will work. That will make the clipboard tests run again in the CI.
There are some tests failing, I guess they use the same shared resources, but are not in the same scope, but the timings look ok.

Comparing times with #28542, those are the results:

| Build | Original | dist=loadscope | diff |
|-------|----------|------------------|-----|
| Linux p35_compat | 25m | 25m | 0 |
| Linux py36_32bit | 25m | 25m | 0 |
| Linux py36_locale_slow | 24m | 26m | +2 |
| Linux py36_locale_slow_old_np | 21m | 26m | +5 |
| Linux py37_locale | 26m | 25m | -1 |
| Linux py37_np_dev | 22m | 23m | +1 |
| Windows py36_np15 | 25m | 26m | +1 |
| Windows py37_np141 | 23m | 22m | -1 |
| macOS py35_macos | 28m | 22m | -6 |
| travis 3.7 | 21m | 23m | +2 |
| travis 3.6, locale | 23m | 25m | +2 |
| travis 3.6, coverage | 28m | 31m | +3 |
| travis 3.6, slow | 24m | 22m | -2 |

I'll have a look at `Linux py36_locale_slow_old_np` to see if the increase on time is likely to be caused by the different parallelization of tests implemented here. And split some test files. Or if it's random noise or something else. But for the rest looks like there is no significant increase in time. So, I think it makes sense to move forward with this.
Looks like in `test_sql.py` there are files in different classes that use the same shared resources, so I changed the distribution of the tests to be per file, and not per scope.

I think we'll have to detect the file(s) that are taking longer in `Linux py36_locale_slow_old_np` and split them. Other than that, this looks good to me (there is an error in the allowed failures in Travis, but I guess it's unrelated).

See updated timings:

| Build | Original | dist=loadscope | diff | dist=loadfile |
|-------|----------|------------------|-----|----------------|
| Linux p35_compat | 25m | 25m | 0 | 24m |
| Linux py36_32bit | 25m | 25m | 0 | 26m |
| Linux py36_locale_slow | 24m | 26m | +2 | 28m |
| Linux py36_locale_slow_old_np | 21m | 26m | +5 | 30m |
| Linux py37_locale | 26m | 25m | -1 | 26m |
| Linux py37_np_dev | 22m | 23m | +1 | 23m |
| Windows py36_np15 | 25m | 26m | +1 | 25m |
| Windows py37_np141 | 23m | 22m | -1 | 23m |
| macOS py35_macos | 28m | 22m | -6 | 21m |
| travis 3.7 | 21m | 23m | +2 | 26m |
| travis 3.6, locale | 23m | 25m | +2 | 26m |
| travis 3.6, coverage | 28m | 31m | +3 | 31m |
| travis 3.6, slow | 24m | 22m | -2 | 27m |
Would be nice to get this merged somehow soon, so we can restore the clipboard tests. But this PR is a bit tricky and somehow risky, so let's review well, and re-run the CI couple of times before merging.
ok one more rebase; ping on green.
Having this error in Travis:
```
QXcbClipboard::setMimeData: Cannot set X11 selection owner

................................................./home/travis/build/pandas-dev/pandas/pandas/compat/__init__.py:85: UserWarning: Could not import the lzma module. Your installed Python is incomplete. Attempting to use lzma compression will result in a RuntimeError.

  warnings.warn(msg)

/home/travis/build/pandas-dev/pandas/pandas/compat/__init__.py:85: UserWarning: Could not import the lzma module. Your installed Python is incomplete. Attempting to use lzma compression will result in a RuntimeError.

  warnings.warn(msg)

./home/travis/build/pandas-dev/pandas/pandas/compat/__init__.py:85: UserWarning: Could not import the lzma module. Your installed Python is incomplete. Attempting to use lzma compression will result in a RuntimeError.

  warnings.warn(msg)

/home/travis/build/pandas-dev/pandas/pandas/compat/__init__.py:85: UserWarning: Could not import the lzma module. Your installed Python is incomplete. Attempting to use lzma compression will result in a RuntimeError.

  warnings.warn(msg)

=================================== FAILURES ===================================

______________________ test_raw_roundtrip[\U0001f44d...] _______________________

[gw0] linux -- Python 3.6.9 /home/travis/miniconda3/envs/pandas-dev/bin/python

data = '👍...'

    @pytest.mark.single

    @pytest.mark.clipboard

    @pytest.mark.skipif(not _DEPS_INSTALLED, reason="clipboard primitives not installed")

    @pytest.mark.parametrize("data", ["\U0001f44d...", "Ωœ∑´...", "abcd..."])

    def test_raw_roundtrip(data):

        # PR #25040 wide unicode wasn't copied correctly on PY3 on windows

        clipboard_set(data)

>       assert data == clipboard_get()

E       AssertionError: assert '👍...' == ''

E         - 👍...

pandas/tests/io/test_clipboard.py:266: AssertionError
```

Not sure why this is failing here. Shouldn't be caused by the changes here. Has anyone seen this error before?
@datapythonista are we actually running the x clip tests currently? i thought was part of the issue
I thought we moved those tests to azure, where they were skipped. Not sure if the failing build was added after that. I'm in my phone, can't check in detail now.
@datapythonista still working on this?
Yes, still need to figure out what's the problem with the failing tests, but want to get this merged.
can you rebase when you have some time
it looks like the 3.6 coverage job is taking 50% longer than on master. can you confirm? (45 min)

maybe rebase and see if this repeats.
thanks @datapythonista 

let's monitor the CI going forward for any odd failures
I created #29671 to speed up the builds.
great

in practice we are already splitting test files if they r too big (in loc)
but this is also a good metric

Potentially related seeing this on travis 36 builds
```
[gw1] linux -- Python 3.6.6 /home/travis/miniconda3/envs/pandas-dev/bin/python
data = '👍...'
    @pytest.mark.single
    @pytest.mark.clipboard
    @pytest.mark.skipif(not _DEPS_INSTALLED, reason="clipboard primitives not installed")
    @pytest.mark.parametrize("data", ["\U0001f44d...", "Ωœ∑´...", "abcd..."])
    def test_raw_roundtrip(data):
        # PR #25040 wide unicode wasn't copied correctly on PY3 on windows
        clipboard_set(data)
>       assert data == clipboard_get()
E       AssertionError: assert '👍...' == ''
E         - 👍...
pandas/tests/io/test_clipboard.py:264: AssertionError
```
https://travis-ci.org/pandas-dev/pandas/jobs/613141276?utm_medium=notification&utm_source=github_status

https://travis-ci.org/pandas-dev/pandas/jobs/613153228?utm_medium=notification&utm_source=github_status