Repository: pandas-dev/pandas
Pull request: CI: Migrate Python 3.10 testing to Posix GHA/Azure Pipelines
- [ ] closes #43890
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Problem statement:
CI: Move Python 3.10 workflows to Posix Matrix File
We should move Python 3.10 workflows from the python-dev workflow to individual jobs. 

The jobs we should add are: Azure 3.10 macOs, Azure 3.10 Windows, Posix jobs(actions-310-slow, actions-310, convert actions-39-numpydev to actions-310-numpydev). Finally, we should change the Python-Dev workflow to be testing the 3.11 alpha.

Feel free to edit if I'm missing any jobs.

Hints from issue discussion:
This needs to be done to clear the way for Python 3.11 dev testing and for good coverage on Python 3.10.
@lithomas1 is this blocking 1.4 release?
It's not blocking for the RC, but blocks the release. OP was a little outdated, so I updated it with what needs to be done.

Commit message: CI: Migrate Python 3.10 testing to Posix GHA/Azure Pipelines

Patch to review:
diff --git a/.github/workflows/python-dev.yml b/.github/workflows/python-dev.yml
index a7c7f3b739f65..d23be2755fd86 100644
--- a/.github/workflows/python-dev.yml
+++ b/.github/workflows/python-dev.yml
@@ -1,92 +1,100 @@
-name: Python Dev
+# This file is purposely frozen. DO NOT DELETE IT
+# Unfreeze once the next Python Dev version has released beta 1
+# and both Cython and numpy support it
+# After that Python has released, migrate the workflows to the
+# posix GHA workflows/Azure pipelines and "freeze" this file by
+# commenting it out.
+# Feel free to modify this comment as necessary.
 
-on:
-  push:
-    branches:
-      - master
-      - 1.3.x
-  pull_request:
-    branches:
-      - master
-      - 1.3.x
-    paths-ignore:
-      - "doc/**"
-
-env:
-  PYTEST_WORKERS: "auto"
-  PANDAS_CI: 1
-  PATTERN: "not slow and not network and not clipboard"
-  COVERAGE: true
-  PYTEST_TARGET: pandas
-
-jobs:
-  build:
-    runs-on: ${{ matrix.os }}
-    strategy:
-      fail-fast: false
-      matrix:
-        os: [ubuntu-latest, macOS-latest, windows-latest]
-
-    name: actions-310-dev
-    timeout-minutes: 80
-
-    concurrency:
-      #https://github.community/t/concurrecy-not-work-for-push/183068/7
-      group: ${{ github.event_name == 'push' && github.run_number || github.ref }}-${{ matrix.os }}-${{ matrix.pytest_target }}-dev
-      cancel-in-progress: true
-
-    steps:
-    - uses: actions/checkout@v2
-      with:
-        fetch-depth: 0
-
-    - name: Set up Python Dev Version
-      uses: actions/setup-python@v2
-      with:
-        python-version: '3.10-dev'
-
-    # TODO: GH#44980 https://github.com/pypa/setuptools/issues/2941
-    - name: Install dependencies
-      shell: bash
-      run: |
-        python -m pip install --upgrade pip "setuptools<60.0.0" wheel
-        pip install -i https://pypi.anaconda.org/scipy-wheels-nightly/simple numpy
-        pip install git+https://github.com/nedbat/coveragepy.git
-        pip install cython python-dateutil pytz hypothesis pytest>=6.2.5 pytest-xdist pytest-cov
-        pip list
-
-    - name: Build Pandas
-      run: |
-        python setup.py build_ext -q -j2
-        python -m pip install -e . --no-build-isolation --no-use-pep517
-
-    - name: Build Version
-      run: |
-        python -c "import pandas; pandas.show_versions();"
-
-    - name: Test with pytest
-      shell: bash
-      run: |
-        ci/run_tests.sh
-
-    - name: Publish test results
-      uses: actions/upload-artifact@master
-      with:
-        name: Test results
-        path: test-data.xml
-      if: failure()
-
-    - name: Print skipped tests
-      run: |
-        python ci/print_skipped.py
-
-    - name: Report Coverage
-      run: |
-        coverage report -m
-
-    - name: Upload coverage to Codecov
-      uses: codecov/codecov-action@v2
-      with:
-        flags: unittests
-        name: codecov-pandas
-        fail_ci_if_error: true
+#name: Python Dev
+#
+#on:
+#  push:
+#    branches:
+#      - master
+#      - 1.3.x
+#  pull_request:
+#    branches:
+#      - master
+#      - 1.3.x
+#    paths-ignore:
+#      - "doc/**"
+#
+#env:
+#  PYTEST_WORKERS: "auto"
+#  PANDAS_CI: 1
+#  PATTERN: "not slow and not network and not clipboard"
+#  COVERAGE: true
+#  PYTEST_TARGET: pandas
+#
+#jobs:
+#  build:
+#    runs-on: ${{ matrix.os }}
+#    strategy:
+#      fail-fast: false
+#      matrix:
+#        os: [ubuntu-latest, macOS-latest, windows-latest]
+#
+#    name: actions-310-dev
+#    timeout-minutes: 80
+#
+#    concurrency:
+#      #https://github.community/t/concurrecy-not-work-for-push/183068/7
+#      group: ${{ github.event_name == 'push' && github.run_number || github.ref }}-${{ matrix.os }}-${{ matrix.pytest_target }}-dev
+#      cancel-in-progress: true
+#
+#    steps:
+#    - uses: actions/checkout@v2
+#      with:
+#        fetch-depth: 0
+#
+#    - name: Set up Python Dev Version
+#      uses: actions/setup-python@v2
+#      with:
+#        python-version: '3.11-dev'
+#
+#    # TODO: GH#44980 https://github.com/pypa/setuptools/issues/2941
+#    - name: Install dependencies
+#      shell: bash
+#      run: |
+#        python -m pip install --upgrade pip "setuptools<60.0.0" wheel
+#        pip install -i https://pypi.anaconda.org/scipy-wheels-nightly/simple numpy
+#        pip install git+https://github.com/nedbat/coveragepy.git
+#        pip install cython python-dateutil pytz hypothesis pytest>=6.2.5 pytest-xdist pytest-cov
+#        pip list
+#
+#    - name: Build Pandas
+#      run: |
+#        python setup.py build_ext -q -j2
+#        python -m pip install -e . --no-build-isolation --no-use-pep517
+#
+#    - name: Build Version
+#      run: |
+#        python -c "import pandas; pandas.show_versions();"
+#
+#    - name: Test with pytest
+#      shell: bash
+#      run: |
+#        ci/run_tests.sh
+#
+#    - name: Publish test results
+#      uses: actions/upload-artifact@master
+#      with:
+#        name: Test results
+#        path: test-data.xml
+#      if: failure()
+#
+#    - name: Print skipped tests
+#      run: |
+#        python ci/print_skipped.py
+#
+#    - name: Report Coverage
+#      run: |
+#        coverage report -m
+#
+#    - name: Upload coverage to Codecov
+#      uses: codecov/codecov-action@v2
+#      with:
+#        flags: unittests
+#        name: codecov-pandas
+#        fail_ci_if_error: true
diff --git a/ci/deps/actions-39-numpydev.yaml b/ci/deps/actions-310-numpydev.yaml
similarity index 95%
rename from ci/deps/actions-39-numpydev.yaml
rename to ci/deps/actions-310-numpydev.yaml
index 4a6acf55e265f..3e32665d5433f 100644
--- a/ci/deps/actions-39-numpydev.yaml
+++ b/ci/deps/actions-310-numpydev.yaml
@@ -2,7 +2,7 @@ name: pandas-dev
 channels:
   - defaults
 dependencies:
-  - python=3.9
+  - python=3.10
 
   # tools
   - pytest>=6.0
diff --git a/ci/deps/actions-310-slow.yaml b/ci/deps/actions-310-slow.yaml
new file mode 100644
index 0000000000000..8ea284d24ed3d
--- /dev/null
+++ b/ci/deps/actions-310-slow.yaml
@@ -0,0 +1,41 @@
+name: pandas-dev
+channels:
+  - defaults
+  - conda-forge
+dependencies:
+  - python=3.10
+
+  # tools
+  - cython>=0.29.24
+  - pytest>=6.0
+  - pytest-cov
+  - pytest-xdist>=1.31
+  - hypothesis>=5.5.3
+
+  # pandas dependencies
+  - beautifulsoup4
+  - bottleneck
+  - boto3
+  - fsspec>=0.8.0
+  - gcsfs
+  - html5lib
+  - jinja2
+  - lxml
+  - matplotlib
+  - numba
+  - numexpr
+  - numpy
+  - openpyxl
+  - pyarrow>2.0.1
+  - pytables
+  - python-dateutil
+  - pytz
+  - s3fs>=0.4.2
+  - scipy
+  - sqlalchemy
+  - xlrd
+  - xlsxwriter
+  - xlwt
+  - pyreadstat
+  - pyxlsb
+  - zstandard
diff --git a/ci/deps/actions-310.yaml b/ci/deps/actions-310.yaml
new file mode 100644
index 0000000000000..6bf5e1ddbebc6
--- /dev/null
+++ b/ci/deps/actions-310.yaml
@@ -0,0 +1,40 @@
+name: pandas-dev
+channels:
+  - conda-forge
+dependencies:
+  - python=3.10
+
+  # tools
+  - cython>=0.29.24
+  - pytest>=6.0
+  - pytest-cov
+  - pytest-xdist>=1.31
+  - hypothesis>=5.5.3
+
+  # pandas dependencies
+  - beautifulsoup4
+  - bottleneck
+  - boto3
+  - fsspec>=0.8.0
+  - gcsfs
+  - html5lib
+  - jinja2
+  - lxml
+  - matplotlib
+  - numpy  # move up to solve before numba
+  - numba
+  - numexpr
+  - openpyxl
+  - pyarrow>2.0.1
+  - pytables
+  - python-dateutil
+  - pytz
+  - s3fs>=0.4.2
+  - scipy
+  - sqlalchemy
+  - xlrd
+  - xlsxwriter
+  - xlwt
+  - pyreadstat
+  - pyxlsb
+  - zstandard
diff --git a/ci/deps/azure-macos-310.yaml b/ci/deps/azure-macos-310.yaml
new file mode 100644
index 0000000000000..eb984df1aa739
--- /dev/null
+++ b/ci/deps/azure-macos-310.yaml
@@ -0,0 +1,36 @@
+name: pandas-dev
+channels:
+  - defaults
+  - conda-forge
+dependencies:
+  - python=3.10
+
+  # tools
+  - cython>=0.29.24
+  - pytest>=6.0
+  - pytest-xdist>=1.31
+  - hypothesis>=5.5.3
+  - pytest-azurepipelines
+
+  # pandas dependencies
+  - beautifulsoup4
+  - bottleneck
+  - html5lib
+  - jinja2
+  - lxml
+  - matplotlib=3.3.2
+  - nomkl
+  - numexpr
+  - numpy=1.21.3
+  - openpyxl
+  - pyarrow=1.0.1
+  - pyreadstat
+  - pytables
+  - python-dateutil==2.8.1
+  - pytz
+  - pyxlsb
+  - xarray
+  - xlrd
+  - xlsxwriter
+  - xlwt
+  - zstandard
diff --git a/ci/deps/azure-macos-38.yaml b/ci/deps/azure-macos-38.yaml
index 472dc8754d13e..422aa86c57fc7 100644
--- a/ci/deps/azure-macos-38.yaml
+++ b/ci/deps/azure-macos-38.yaml
@@ -6,6 +6,7 @@ dependencies:
   - python=3.8
 
   # tools
+  - cython>=0.29.24
   - pytest>=6.0
   - pytest-xdist>=1.31
   - hypothesis>=5.5.3
@@ -33,6 +34,3 @@ dependencies:
   - xlsxwriter
   - xlwt
   - zstandard
-  - pip
-  - pip:
-    - cython>=0.29.24
diff --git a/ci/deps/azure-macos-39.yaml b/ci/deps/azure-macos-39.yaml
new file mode 100644
index 0000000000000..a0860ef536953
--- /dev/null
+++ b/ci/deps/azure-macos-39.yaml
@@ -0,0 +1,36 @@
+name: pandas-dev
+channels:
+  - defaults
+  - conda-forge
+dependencies:
+  - python=3.9
+
+  # tools
+  - cython>=0.29.24
+  - pytest>=6.0
+  - pytest-xdist>=1.31
+  - hypothesis>=5.5.3
+  - pytest-azurepipelines
+
+  # pandas dependencies
+  - beautifulsoup4
+  - bottleneck
+  - html5lib
+  - jinja2
+  - lxml
+  - matplotlib=3.3.2
+  - nomkl
+  - numexpr
+  - numpy=1.21.3
+  - openpyxl
+  - pyarrow=1.0.1
+  - pyreadstat
+  - pytables
+  - python-dateutil==2.8.1
+  - pytz
+  - pyxlsb
+  - xarray
+  - xlrd
+  - xlsxwriter
+  - xlwt
+  - zstandard
diff --git a/ci/deps/azure-windows-310.yaml b/ci/deps/azure-windows-310.yaml
new file mode 100644
index 0000000000000..ccbae9afcdd38
--- /dev/null
+++ b/ci/deps/azure-windows-310.yaml
@@ -0,0 +1,40 @@
+name: pandas-dev
+channels:
+  - conda-forge
+  - defaults
+dependencies:
+  - python=3.10
+
+  # tools
+  - cython>=0.29.24
+  - pytest>=6.0
+  - pytest-xdist>=1.31
+  - hypothesis>=5.5.3
+  - pytest-azurepipelines
+
+  # pandas dependencies
+  - beautifulsoup4
+  - bottleneck
+  - fsspec>=0.8.0
+  - gcsfs
+  - html5lib
+  - jinja2
+  - lxml
+  - matplotlib
+  - numba
+  - numexpr
+  - numpy
+  - openpyxl
+  - pyarrow
+  - pytables
+  - python-dateutil
+  - pytz
+  - s3fs>=0.4.2
+  - scipy
+  - sqlalchemy
+  - xlrd
+  - xlsxwriter
+  - xlwt
+  - pyreadstat
+  - pyxlsb
+  - zstandard


Code review comments:
diff_hunk: @@ -1,92 +1,100 @@
-name: Python Dev
+# This file is purposely frozen. DO NOT DELETE IT
+# Unfreeze once the next Python Dev version has released beta 1
+# and both Cython and numpy support it
+# After that Python has released, migrate the workflows to the
+# posix GHA workflows/Azure pipelines and "freeze" this file by
+# commenting it out.
+# Feel free to modify this comment as necessary.
 
-on:
-  push:
-    branches:
-      - master
-      - 1.3.x
-  pull_request:
-    branches:
-      - master
-      - 1.3.x
-    paths-ignore:
-      - "doc/**"
-
-env:
-  PYTEST_WORKERS: "auto"
-  PANDAS_CI: 1
-  PATTERN: "not slow and not network and not clipboard"
-  COVERAGE: true
-  PYTEST_TARGET: pandas
-
-jobs:
-  build:
path: .github/workflows/python-dev.yml
text: @user1:
Could also freeze by specifying `if: false`?

@user1:
Nit 2 other things:

`name: actions-310-dev` maybe should be `name: actions-311-dev`?

`branches: - 1.3.x` -> `- 1.4.x`