Repository: pandas-dev/pandas
Pull request: DEPS: Unifying testing and building dependencies across builds
- [X] closes #29664

In our CI builds we're using different versions of pytest, pytest-xdist and other tools. Looks like Python 3.5 was forcing part of this, since some packages were not available.

In this PR I standardize the tools and versions we use in all packages, with only two exceptions:
- In 32 bits we install `Cython` from pip instead of conda, since conda doesn't have the version we want
- `pytest-azurepipelines` is installed only in builds currently running in pipelines

I also reorganize a bit the builds so dependencies are easier to find. First is always Python, then the block with the tools, which is always the same with the exceptions mentioned above, and finally the dependencies.

Problem statement:
CI: Unify pytest versions across builds
May be I'm missing something, but is there a reason to use different pytest versions in our CI builds? I see we're using those in the dependencies files:
- pytest
- pytest>=4.0.2
- pytest>=4.0.2,<5.0
- pytest=4.5.0
- pytest>=5.0.0
- pytest>=5.0.1

Since pytest is not a pandas dependency, but just a tool we use, feels like we should just use always the same version (the latest). All those (all versions since pytest 4) support Python >3.6, so I don't think there is any limitation that prevents this.

Am I missing something? Can we always use the same version?

I'm thinking on having a separate file for the tools needed for all builds (pytest, pytest-xdist...), and install both dependency files, the one of the build, and the common one. So we don't have to duplicate those in every file (it's faster to maintain and make changes, and also easier to see what dependencies we have and what we're testing in every build).

@jreback @TomAugspurger does this make sense?

Hints from issue discussion:
you can try but we had a number of issues and that is why they r pinned
-1 on splitting the build files 
take
lol
it worked!
we need an auto unassigner maybe 2 weeks after no action 
I agree, I'll probably wait until we have unassigned issues with no activity. Having examples will probably help make better decisions, on timing, what actions we consider...

Commit message: Adding pytest-cov, deleted by mistake

Patch to review:
diff --git a/ci/deps/azure-36-32bit.yaml b/ci/deps/azure-36-32bit.yaml
index 1e2e6c33e8c15..30a9da553f897 100644
--- a/ci/deps/azure-36-32bit.yaml
+++ b/ci/deps/azure-36-32bit.yaml
@@ -3,21 +3,23 @@ channels:
   - defaults
   - conda-forge
 dependencies:
+  - python=3.6.*
+  # tools
+  # - cython=0.29.13 ### Cython 0.29.13 for 32 bits is not available with conda, installing below with pip instead
+  - pytest
+  - pytest-xdist>=1.21
+  - pytest-mock
+  - hypothesis>=3.58.0
+  - pytest-azurepipelines
+  # pandas dependencies
   - attrs=19.1.0
   - gcc_linux-32
-  - gcc_linux-32
   - gxx_linux-32
   - numpy=1.14.*
   - python-dateutil
-  - python=3.6.*
   - pytz=2017.2
-  # universal
-  - pytest
-  - pytest-xdist
-  - pytest-mock
-  - pytest-azurepipelines
-  - hypothesis>=3.58.0
+
+  # see comment above
   - pip
   - pip:
-    # Anaconda doesn't build a new enough Cython
-    - cython>=0.29.13
+    - cython==0.29.13
diff --git a/ci/deps/azure-36-locale.yaml b/ci/deps/azure-36-locale.yaml
index 76868f598f11b..182e4c2ed8812 100644
--- a/ci/deps/azure-36-locale.yaml
+++ b/ci/deps/azure-36-locale.yaml
@@ -3,28 +3,29 @@ channels:
   - defaults
   - conda-forge
 dependencies:
+  - python=3.6.*
+  # tools
+  - cython=0.29.13
+  - pytest
+  - pytest-xdist>=1.21
+  - pytest-mock
+  - hypothesis>=3.58.0
+  - pytest-azurepipelines
+  # pandas dependencies
   - beautifulsoup4==4.6.0
   - bottleneck=1.2.*
-  - cython=0.29.13
   - lxml
   - matplotlib=2.2.2
   - numpy=1.14.*
   - openpyxl=2.4.8
   - python-dateutil
   - python-blosc
-  - python=3.6.*
   - pytz=2017.2
   - scipy
   - sqlalchemy=1.1.4
   - xlrd=1.1.0
   - xlsxwriter=0.9.8
   - xlwt=1.2.0
-  # universal
-  - pytest>=5.0.0
-  - pytest-xdist>=1.29.0
-  - pytest-mock
-  - pytest-azurepipelines
-  - hypothesis>=3.58.0
   - pip
   - pip:
     - html5lib==1.0b2
diff --git a/ci/deps/azure-36-locale_slow.yaml b/ci/deps/azure-36-locale_slow.yaml
index 21205375204dc..88b81d5c1e42d 100644
--- a/ci/deps/azure-36-locale_slow.yaml
+++ b/ci/deps/azure-36-locale_slow.yaml
@@ -3,8 +3,16 @@ channels:
   - defaults
   - conda-forge
 dependencies:
+  - python=3.6.*
+  # tools
+  - cython=0.29.13
+  - pytest
+  - pytest-xdist>=1.21
+  - pytest-mock
+  - hypothesis>=3.58.0
+  - pytest-azurepipelines
+  # pandas dependencies
   - beautifulsoup4
-  - cython>=0.29.13
   - gcsfs
   - html5lib
   - ipython
@@ -17,7 +25,6 @@ dependencies:
   - openpyxl
   - pytables
   - python-dateutil
-  - python=3.6.*
   - pytz
   - s3fs
   - scipy
@@ -25,12 +32,4 @@ dependencies:
   - xlrd
   - xlsxwriter
   - xlwt
-  # universal
-  - pytest>=4.0.2
-  - pytest-xdist
-  - pytest-mock
-  - pytest-azurepipelines
   - moto
-  - pip
-  - pip:
-    - hypothesis>=3.58.0
diff --git a/ci/deps/azure-36-minimum_versions.yaml b/ci/deps/azure-36-minimum_versions.yaml
index e2c78165fe4b9..6f451ea85ff2e 100644
--- a/ci/deps/azure-36-minimum_versions.yaml
+++ b/ci/deps/azure-36-minimum_versions.yaml
@@ -3,25 +3,26 @@ channels:
   - defaults
   - conda-forge
 dependencies:
+  - python=3.6.1
+  # tools
+  - cython=0.29.13
+  - pytest
+  - pytest-xdist>=1.21
+  - pytest-mock
+  - hypothesis>=3.58.0
+  - pytest-azurepipelines
+  # pandas dependencies
   - beautifulsoup4=4.6.0
   - bottleneck=1.2.1
-  - cython>=0.29.13
   - jinja2=2.8
   - numexpr=2.6.2
   - numpy=1.13.3
   - openpyxl=2.4.8
   - pytables=3.4.2
   - python-dateutil=2.6.1
-  - python=3.6.1
   - pytz=2017.2
   - scipy=0.19.0
   - xlrd=1.1.0
   - xlsxwriter=0.9.8
   - xlwt=1.2.0
-  # universal
   - html5lib=1.0.1
-  - hypothesis>=3.58.0
-  - pytest=4.5.0
-  - pytest-xdist
-  - pytest-mock
-  - pytest-azurepipelines
diff --git a/ci/deps/azure-37-locale.yaml b/ci/deps/azure-37-locale.yaml
index 24464adb74f5b..a250781331e25 100644
--- a/ci/deps/azure-37-locale.yaml
+++ b/ci/deps/azure-37-locale.yaml
@@ -3,8 +3,16 @@ channels:
   - defaults
   - conda-forge
 dependencies:
+  - python=3.7.*
+  # tools
+  - cython=0.29.13
+  - pytest
+  - pytest-xdist>=1.21
+  - pytest-mock
+  - hypothesis>=3.58.0
+  - pytest-azurepipelines
+  # pandas dependencies
   - beautifulsoup4
-  - cython>=0.29.13
   - html5lib
   - ipython
   - jinja2
@@ -17,7 +25,6 @@ dependencies:
   - openpyxl
   - pytables
   - python-dateutil
-  - python=3.7.*
   - pytz
   - s3fs
   - scipy
@@ -25,11 +32,3 @@ dependencies:
   - xlrd
   - xlsxwriter
   - xlwt
-  # universal
-  - pytest>=5.0.1
-  - pytest-xdist>=1.29.0
-  - pytest-mock
-  - pytest-azurepipelines
-  - pip
-  - pip:
-    - hypothesis>=3.58.0
diff --git a/ci/deps/azure-37-numpydev.yaml b/ci/deps/azure-37-numpydev.yaml
index 0fb06fd43724c..ebba388d99908 100644
--- a/ci/deps/azure-37-numpydev.yaml
+++ b/ci/deps/azure-37-numpydev.yaml
@@ -3,14 +3,15 @@ channels:
   - defaults
 dependencies:
   - python=3.7.*
-  - pytz
-  - Cython>=0.29.13
-  # universal
-  # pytest < 5 until defaults has pytest-xdist>=1.29.0
-  - pytest>=4.0.2,<5.0
-  - pytest-xdist
+  # tools
+  - cython=0.29.13
+  - pytest
+  - pytest-xdist>=1.21
   - pytest-mock
   - hypothesis>=3.58.0
+  - pytest-azurepipelines
+  # pandas dependencies
+  - pytz
   - pip
   - pip:
     - "git+git://github.com/dateutil/dateutil.git"
@@ -18,5 +19,3 @@ dependencies:
     - "--pre"
     - "numpy"
     - "scipy"
-    # https://github.com/pandas-dev/pandas/issues/27421
-    - pytest-azurepipelines<1.0.0
diff --git a/ci/deps/azure-macos-36.yaml b/ci/deps/azure-macos-36.yaml
index 85c090bf6f938..7c4acd6bcf355 100644
--- a/ci/deps/azure-macos-36.yaml
+++ b/ci/deps/azure-macos-36.yaml
@@ -2,6 +2,15 @@ name: pandas-dev
 channels:
   - defaults
 dependencies:
+  - python=3.6.*
+  # tools
+  - cython=0.29.13
+  - pytest
+  - pytest-xdist>=1.21
+  - pytest-mock
+  - hypothesis>=3.58.0
+  - pytest-azurepipelines
+  # pandas dependencies
   - beautifulsoup4
   - bottleneck
   - html5lib
@@ -14,7 +23,6 @@ dependencies:
   - openpyxl
   - pyarrow
   - pytables
-  - python=3.6.*
   - python-dateutil==2.6.1
   - pytz
   - xarray
@@ -23,13 +31,4 @@ dependencies:
   - xlwt
   - pip
   - pip:
-    # Anaconda / conda-forge don't build for 3.5
-    - cython>=0.29.13
     - pyreadstat
-    # universal
-    - pytest>=5.0.1
-    - pytest-xdist>=1.29.0
-    - pytest-mock
-    - hypothesis>=3.58.0
-    # https://github.com/pandas-dev/pandas/issues/27421
-    - pytest-azurepipelines<1.0.0
diff --git a/ci/deps/azure-windows-36.yaml b/ci/deps/azure-windows-36.yaml
index 88b38aaef237c..1982b7e3519d8 100644
--- a/ci/deps/azure-windows-36.yaml
+++ b/ci/deps/azure-windows-36.yaml
@@ -3,6 +3,15 @@ channels:
   - conda-forge
   - defaults
 dependencies:
+  - python=3.6.*
+  # tools
+  - cython=0.29.13
+  - pytest
+  - pytest-xdist>=1.21
+  - pytest-mock
+  - hypothesis>=3.58.0
+  - pytest-azurepipelines
+  # pandas dependencies
   - blosc
   - bottleneck
   - fastparquet>=0.2.1
@@ -13,16 +22,8 @@ dependencies:
   - pyarrow
   - pytables
   - python-dateutil
-  - python=3.6.*
   - pytz
   - scipy
   - xlrd
   - xlsxwriter
   - xlwt
-  # universal
-  - cython>=0.29.13
-  - pytest>=5.0.1
-  - pytest-xdist>=1.29.0
-  - pytest-mock
-  - pytest-azurepipelines
-  - hypothesis>=3.58.0
diff --git a/ci/deps/azure-windows-37.yaml b/ci/deps/azure-windows-37.yaml
index 7680ed9fd9c92..32303c1186381 100644
--- a/ci/deps/azure-windows-37.yaml
+++ b/ci/deps/azure-windows-37.yaml
@@ -3,6 +3,15 @@ channels:
   - defaults
   - conda-forge
 dependencies:
+  - python=3.7.*
+  # tools
+  - cython=0.29.13
+  - pytest
+  - pytest-xdist>=1.21
+  - pytest-mock
+  - hypothesis>=3.58.0
+  - pytest-azurepipelines
+  # pandas dependencies
   - beautifulsoup4
   - bottleneck
   - gcsfs
@@ -15,7 +24,6 @@ dependencies:
   - numpy=1.14.*
   - openpyxl
   - pytables
-  - python=3.7.*
   - python-dateutil
   - pytz
   - s3fs
@@ -24,11 +32,4 @@ dependencies:
   - xlrd
   - xlsxwriter
   - xlwt
-  # universal
-  - cython>=0.29.13
-  - pytest>=5.0.0
-  - pytest-xdist>=1.29.0
-  - pytest-mock
-  - pytest-azurepipelines
-  - hypothesis>=3.58.0
   - pyreadstat
diff --git a/ci/deps/travis-36-cov.yaml b/ci/deps/travis-36-cov.yaml
index b2a74fceaf0fa..bd6007a62f9ba 100644
--- a/ci/deps/travis-36-cov.yaml
+++ b/ci/deps/travis-36-cov.yaml
@@ -3,6 +3,15 @@ channels:
   - defaults
   - conda-forge
 dependencies:
+  - python=3.6.*
+  # tools
+  - cython=0.29.13
+  - pytest
+  - pytest-xdist>=1.21
+  - pytest-mock
+  - hypothesis>=3.58.0
+  - pytest-cov  # this is only needed in the coverage build
+  # pandas dependencies
   - beautifulsoup4
   - botocore>=1.11
   - cython>=0.29.13
@@ -27,7 +36,6 @@ dependencies:
   - pymysql
   - pytables
   - python-snappy
-  - python=3.6.*
   - pytz
   - s3fs
   - scikit-learn
@@ -38,12 +46,6 @@ dependencies:
   - xlrd
   - xlsxwriter
   - xlwt
-  # universal
-  - pytest>=5.0.1
-  - pytest-xdist>=1.29.0
-  - pytest-cov
-  - pytest-mock
-  - hypothesis>=3.58.0
   - pip
   - pip:
     - brotlipy
diff --git a/ci/deps/travis-36-locale.yaml b/ci/deps/travis-36-locale.yaml
index 09f72e65098c9..1b2e972ed905c 100644
--- a/ci/deps/travis-36-locale.yaml
+++ b/ci/deps/travis-36-locale.yaml
@@ -3,10 +3,17 @@ channels:
   - defaults
   - conda-forge
 dependencies:
+  - python=3.6.*
+  # tools
+  - cython=0.29.13
+  - pytest
+  - pytest-xdist>=1.21
+  - pytest-mock
+  - hypothesis>=3.58.0
+  # pandas dependencies
   - beautifulsoup4
   - blosc=1.14.3
   - python-blosc
-  - cython>=0.29.13
   - fastparquet=0.2.1
   - gcsfs=0.2.2
   - html5lib
@@ -24,7 +31,6 @@ dependencies:
   - pymysql=0.7.11
   - pytables
   - python-dateutil
-  - python=3.6.*
   - pytz
   - s3fs=0.3.0
   - scipy
@@ -33,10 +39,3 @@ dependencies:
   - xlrd
   - xlsxwriter
   - xlwt
-  # universal
-  - pytest>=5.0.1
-  - pytest-xdist>=1.29.0
-  - pytest-mock
-  - pip
-  - pip:
-    - hypothesis>=3.58.0
diff --git a/ci/deps/travis-36-slow.yaml b/ci/deps/travis-36-slow.yaml
index e9c5dadbc924a..215225419bcf3 100644
--- a/ci/deps/travis-36-slow.yaml
+++ b/ci/deps/travis-36-slow.yaml
@@ -3,8 +3,15 @@ channels:
   - defaults
   - conda-forge
 dependencies:
+  - python=3.6.*
+  # tools
+  - cython=0.29.13
+  - pytest
+  - pytest-xdist>=1.21
+  - pytest-mock
+  - hypothesis>=3.58.0
+  # pandas dependencies
   - beautifulsoup4
-  - cython>=0.29.13
   - html5lib
   - lxml
   - matplotlib
@@ -16,7 +23,6 @@ dependencies:
   - pymysql
   - pytables
   - python-dateutil
-  - python=3.6.*
   - pytz
   - s3fs
   - scipy
@@ -24,9 +30,4 @@ dependencies:
   - xlrd
   - xlsxwriter
   - xlwt
-  # universal
-  - pytest>=5.0.0
-  - pytest-xdist>=1.29.0
-  - pytest-mock
   - moto
-  - hypothesis>=3.58.0
diff --git a/ci/deps/travis-37.yaml b/ci/deps/travis-37.yaml
index 903636f2fe060..acc57f9e6b6d2 100644
--- a/ci/deps/travis-37.yaml
+++ b/ci/deps/travis-37.yaml
@@ -5,20 +5,21 @@ channels:
   - c3i_test
 dependencies:
   - python=3.7.*
+  # tools
+  - cython=0.29.13
+  - pytest
+  - pytest-xdist>=1.21
+  - pytest-mock
+  - hypothesis>=3.58.0
+  # pandas dependencies
   - botocore>=1.11
-  - cython>=0.29.13
   - numpy
   - python-dateutil
   - nomkl
   - pyarrow
   - pytz
-  # universal
-  - pytest>=5.0.0
-  - pytest-xdist>=1.29.0
-  - pytest-mock
-  - hypothesis>=3.58.0
   - s3fs
-  - pip
   - pyreadstat
+  - pip
   - pip:
     - moto


Code review comments:
diff_hunk: @@ -3,8 +3,16 @@ channels:
   - defaults
   - conda-forge
 dependencies:
+  - python=3.7.*
+  # tools
+  - cython=0.29.13
path: ci/deps/azure-37-locale.yaml
text: @user1:
these should be >=

diff_hunk: @@ -3,8 +3,16 @@ channels:
   - defaults
   - conda-forge
 dependencies:
+  - python=3.6.*
+  # tools
+  - cython=0.29.13
+  - pytest
path: ci/deps/azure-36-locale_slow.yaml
text: @user1:
need a min pin on pytest (for all .yaml)

@author:
We've got all these versions in the builds:
- pytest
- pytest>=4.0.2
- pytest>=4.0.2,<5.0
- pytest=4.5.0
- pytest>=5.0.0
- pytest>=5.0.1

Any preference on which you want to require? `pytest>-5.0.1` is ok?

@user1:
yeah ok with the >=4.0.2 I think < 4 we don't support at all

diff_hunk: @@ -3,20 +3,19 @@ channels:
   - defaults
 dependencies:
   - python=3.7.*
-  - pytz
-  - Cython>=0.29.13
-  # universal
-  # pytest < 5 until defaults has pytest-xdist>=1.29.0
-  - pytest>=4.0.2,<5.0
-  - pytest-xdist
+  # tools
+  - cython=0.29.13
path: ci/deps/azure-37-numpydev.yaml
text: @user1:
same (and all others except for the 3.6-min build can have >= for cython)

@author:
In the minimum versions build we have a `>=0.29.13` now. There is a `=0.29.13` in another build, not sure if that was intentional.

I guess if we pin cython, when this causes a failure, the immediate action will be to pin it to something bigger where it works. So, is it really worth forcing in a build the `0.29.3` version?