DEPS: Unifying testing and building dependencies across builds
State: closed

- [X] closes #29664

In our CI builds we're using different versions of pytest, pytest-xdist and other tools. Looks like Python 3.5 was forcing part of this, since some packages were not available.

In this PR I standardize the tools and versions we use in all packages, with only two exceptions:
- In 32 bits we install `Cython` from pip instead of conda, since conda doesn't have the version we want
- `pytest-azurepipelines` is installed only in builds currently running in pipelines

I also reorganize a bit the builds so dependencies are easier to find. First is always Python, then the block with the tools, which is always the same with the exceptions mentioned above, and finally the dependencies.

Labels:
name: CI
description: Continuous Integration
name: Dependencies
description: Required and optional dependencies

Comments:
@jreback Looks like all the reasons for the different versions are now gone, except the one for 32 bits (which will still use the same version, but from pip instead of conda)
I addressed the comments.

My understanding (and I can surely be missing something) is that we shouldn't use the same rationale for the dependencies and our tools.

For dependencies (e.g. numpy), we want to make sure that pandas works with 1.13.3 and above, so we test that version, the latest, and probably something in between. Users installing pandas that want a specific version can do that, as far as it's supported.

But for tools (e.g. pytest), we don't care about users in terms of versions. It's something we use as developers, and users don't install or care about. For what I know, I think it's common to just decide a version (the latest unless there is a reason to use another), we all use the same, and we don't care about compatibility... Pinning a specific version would make conda solver life much easier (faster), but we'd need to manually keep increasing it if we don't want to end up with old versions. So, I guess no version or a `>=` is a good option too.

I'd say cython is more in the tools side, but users can compile pandas themselves, so we can consider supporting more than a version. But I think it's also reasonable to say that pandas compiles with Cython version X, and not bother on supporting multiple versions for something where it probably doesn't add any value. Users may have code in numpy 1.13 and the latest pandas, and we want to let them run it. But I don't see the advantage of telling users they can compile pandas with Cython 0.28 and 0.29. They should compile in 0.29 since the generated code should be better.
@datapythonista we DO care about the tools versions. Having tool much churn on these is bad. Other things being equal, we also want as much compatibility as possible.

We like to keep things as stable as possible; when you are a developer and come back to a project, having everything break is a bad developer experience. 

So this is a balance, between using the later-latest tools and keeping some compatibility.

We are already having a lot of churn with black/mypy moving pretty fast, so changing the world otherwise is not great.
For cython we have very specific minimum versions, you *cannot* use it with earlier versions. Similar to the other tools, we try not to move this too often.
For pytest I care less (using the latest on CI is good for me), but for cython I think we should have a build that pins to the oldest supported (which is often very recent anyway). 
We have specific version checks and messaging about the supported cython version in setup.py, so I think this is something we should test it actually works.
I also don't care about the pytest version.
@datapythonista can you rebase (as added the 3.8 build).

also I would like to fix pytest on the 3.6 min build to what we say the min is (4.02 or 5.01) whatever works. 

cython ok with fixing that on the oldest build, on the others >= is ok.
Thanks for the comments. I'll leave the pytest upgrade for a follow up, so we can have a discussion about the version we want first.
Tests actually fail with pytest 4.0.2. They should work with 4.5 (we have a build with it), but I upgraded the versions everywhere to require 5.0.1.
thanks @datapythonista 

as a followup can you update docs in places where we specify pytest version (contributing guide maybe, though I think might be a few places, including show_versions()).

its a develop dependency but users can also run tests with it. this may warrant a release note as well.