CI: limit pytest version on 3.6
State: closed

closes #27406

Labels:
name: CI
description: Continuous Integration

Comments:
Any idea what the root cause is?
we had set this previously on some of the 3.6 builds, no idea really why its actually failing.
@TomAugspurger https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=14549 is failing the numpydev build even with an older pytest; so this is issue is solved for the 3.6 builds, but any idea about this ?

The primary difference is that the numpydev build uses `-W error`. Do you think that the warnings

```
sys:1: ResourceWarning: unclosed file <_io.TextIOWrapper name=0 mode='r' encoding='UTF-8'>
sys:1: ResourceWarning: unclosed file <_io.TextIOWrapper name=0 mode='r' encoding='UTF-8'>
```

are being elevated to errors, causing some internal purest error? I'm not sure where they're coming from, but perhaps adding a `-v` to print out each test name would help.
Not at a computer to check but are those warnings coming from the IO Excel tests? Think I’ve seen them before
Probably not excel, unless they open files before being skipped. That job has pretty minimal dependencies.

I'm not sure if the INTERTNALERROR from pytest is printed inline with the test output, or whether it's buffered till the end :/

```
[gw1] PASSED pandas/tests/arrays/sparse/test_arithmetics.py::TestSparseSeriesArithmetic::test_float_scalar[integer-True-rfloordiv-2-1] 
[gw0] XFAIL pandas/tests/extension/test_categorical.py::TestArithmeticOps::test_arith_frame_with_scalar[__add__] 
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/_pytest/main.py", line 213, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/_pytest/main.py", line 257, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/pluggy/hooks.py", line 289, in __call__
INTERNALERROR>     return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>   File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/pluggy/manager.py", line 87, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/pluggy/manager.py", line 81, in <lambda>
INTERNALERROR>     firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
INTERNALERROR>   File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/pluggy/callers.py", line 208, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/pluggy/callers.py", line 80, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/xdist/dsession.py", line 115, in pytest_runtestloop
INTERNALERROR>     self.loop_once()
INTERNALERROR>   File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/xdist/dsession.py", line 138, in loop_once
INTERNALERROR>     call(**kwargs)
INTERNALERROR>   File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/xdist/dsession.py", line 249, in worker_testreport
INTERNALERROR>     self.config.hook.pytest_runtest_logreport(report=rep)
INTERNALERROR>   File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/pluggy/hooks.py", line 289, in __call__
INTERNALERROR>     return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>   File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/pluggy/manager.py", line 87, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/pluggy/manager.py", line 81, in <lambda>
INTERNALERROR>     firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
INTERNALERROR>   File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/pluggy/callers.py", line 208, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/pluggy/callers.py", line 80, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/pytest_nunit/plugin.py", line 241, in pytest_runtest_logreport
INTERNALERROR>     reporter.record_testreport(report)
INTERNALERROR>   File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/pytest_nunit/plugin.py", line 104, in record_testreport
INTERNALERROR>     if len(testreport.longrepr) > 2:
INTERNALERROR> TypeError: object of type 'ReprExceptionInfo' has no len()
sys:1: ResourceWarning: unclosed file <_io.TextIOWrapper name=0 mode='r' encoding='UTF-8'>
sys:1: ResourceWarning: unclosed file <_io.TextIOWrapper name=0 mode='r' encoding='UTF-8'>
Traceback (most recent call last):
  File "/home/vsts/miniconda3/envs/pandas-dev/bin/pytest", line 11, in <module>
    sys.exit(main())
  File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/_pytest/config/__init__.py", line 74, in main
    return config.hook.pytest_cmdline_main(config=config)
  File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/pluggy/hooks.py", line 289, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/pluggy/manager.py", line 87, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/pluggy/manager.py", line 81, in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/pluggy/callers.py", line 208, in _multicall
    return outcome.get_result()
  File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/pluggy/callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/_pytest/main.py", line 250, in pytest_cmdline_main
    return wrap_session(config, _main)
  File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/_pytest/main.py", line 243, in wrap_session
    session=session, exitstatus=session.exitstatus
  File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/pluggy/hooks.py", line 289, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/pluggy/manager.py", line 87, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/pluggy/manager.py", line 81, in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/pluggy/callers.py", line 203, in _multicall
    gen.send(outcome)
  File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/_pytest/terminal.py", line 649, in pytest_sessionfinish
    outcome.get_result()
  File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/pluggy/callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/pytest_nunit/plugin.py", line 276, in pytest_sessionfinish
    list(case for case in self.cases.values() if case["outcome"] == "passed")
  File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/pytest_nunit/plugin.py", line 276, in <genexpr>
    list(case for case in self.cases.values() if case["outcome"] == "passed")
KeyError: 'outcome'
```
Not sure this is related but comparing the environments between the broken builds and the previous working ones there's a diff in bzip2 version, with it previously being 1.0.7 on the working ones and 1.0.8 on the broken ones. Didn't see another difference save pandas
Sorry can ignore previous comment for now; I thought still looking at Travis failures but those are just timeouts now due to verbosity. Looking at np37 too to see if there's anything that sticks out
this works for me on a local build. (on mac), so must be a different package picked up on linux.
The only diff in deps https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=14539 and https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=14544 is pytest-azure-pipelines. Going to try pinning that to the old value of 0.8.0.
Oh, sorry two different. The failing build also has pytest-nunit (maybe a pytest-azure-pipelines dep?)
thanks @TomAugspurger