Repository: pymc-devs/pymc
Pull request: Combine test workflows into one
This PR combines `pytest`, `arviz-compat` and `jaxtests` workflows into one (big) `tests` workflow.

# Motivations
* Cancelling all workflows is now 3x less clicking. (Only core devs have the necesary permissions to cancel workflows.)
* GitHub Actions workflows can't depend on each other, but if we want to fail fast on ❌ `pre-commit` (#5041), having all jobs in one workflows will make it a lot easier.
* Tracking workflow runtime via https://share.streamlit.io/koaning/gitlit/main will be easier after they're all in one.

# Changes
* ✔ Removed the `buildosx` build script. Chris confirmed via Slack that it can be removed.
* ✔ `arviz-compat` and `jaxtests` were deleted - their jobs were moved into `tests` (previously `pytest`.
* ✔ Upgraded the tests from `ubuntu 18.04` to `20.04` which [is the most recent LTS release](https://wiki.ubuntu.com/Releases).
* ✔ Redistributed tests between jobs to reduce the runtime (closes #5475)

Problem statement:
Combine windows workflows
We have 4 Windows workflows with a combined test runtime of \~10 minutes, but we are spending about 4x5~20 minutes just building the conda environments

If we can combine the tests in 1/2 workflows we could save some compute time without prolonging the total runtimes

Hints from issue discussion:

Commit message: Combine test workflows

The steps within each job were not changed in this commit.

Patch to review:
diff --git a/.github/workflows/arviz_compat.yml b/.github/workflows/arviz_compat.yml
deleted file mode 100644
index e12ddba664..0000000000
--- a/.github/workflows/arviz_compat.yml
+++ /dev/null
@@ -1,100 +0,0 @@
-name: arviz-compatibility
-
-on:
-  pull_request:
-    paths:
-      - ".github/workflows/*.yml"
-      - "pymc/**.py"
-      - "*.py"
-      - "conda-envs/**"
-      - "codecov.yml"
-      - "scripts/*.sh"
-  push:
-    branches: [main]
-    paths:
-      - ".github/workflows/*.yml"
-      - "pymc/**.py"
-      - "*.py"
-      - "conda-envs/**"
-      - "codecov.yml"
-      - "scripts/*.sh"
-
-jobs:
-  pytest:
-    strategy:
-      matrix:
-        os: [ubuntu-latest, macos-latest]
-        floatx: [float64]
-        test-subset:
-          - |
-            pymc/tests/test_parallel_sampling.py
-            pymc/tests/test_posteriors.py
-            pymc/tests/test_sampling.py
-
-          - |
-            pymc/tests/test_data_container.py
-            pymc/tests/test_idata_conversion.py
-            pymc/tests/test_missing.py
-            pymc/tests/test_model.py
-            pymc/tests/test_shape_handling.py
-            pymc/tests/test_shared.py
-      fail-fast: false
-    runs-on: ${{ matrix.os }}
-    env:
-      TEST_SUBSET: ${{ matrix.test-subset }}
-      AESARA_FLAGS: floatX=${{ matrix.floatx }},gcc__cxxflags='-march=native'
-    defaults:
-      run:
-        shell: bash -l {0}
-    steps:
-      - uses: actions/checkout@v2
-      - name: Cache conda
-        uses: actions/cache@v1
-        env:
-          # Increase this value to reset cache if environment-test-py39.yml has not changed
-          CACHE_NUMBER: 0
-        with:
-          path: ~/conda_pkgs_dir
-          key: ${{ runner.os }}-conda-${{ env.CACHE_NUMBER }}-${{
-            hashFiles('conda-envs/environment-test-py39.yml') }}
-      - name: Cache multiple paths
-        uses: actions/cache@v2
-        env:
-          # Increase this value to reset cache if requirements.txt has not changed
-          CACHE_NUMBER: 0
-        with:
-          path: |
-            ~/.cache/pip
-            $RUNNER_TOOL_CACHE/Python/*
-            ~\AppData\Local\pip\Cache
-          key: ${{ runner.os }}-build-${{ matrix.python-version }}-${{
-            hashFiles('requirements.txt') }}
-      - uses: conda-incubator/setup-miniconda@v2
-        with:
-          miniforge-variant: Mambaforge
-          miniforge-version: latest
-          mamba-version: "*"
-          activate-environment: pymc-test-py39
-          channel-priority: strict
-          environment-file: conda-envs/environment-test-py39.yml
-          use-mamba: true
-          use-only-tar-bz2: true # IMPORTANT: This needs to be set for caching to work properly!
-      - name: Install pymc
-        run: |
-          conda activate pymc-test-py39
-          pip install -e .
-          python --version
-      - name: Install latest arviz
-        run: |
-          conda activate pymc-test-py39
-          pip uninstall arviz -y
-          pip install git+https://github.com/arviz-devs/arviz
-      - name: Run tests
-        run: |
-          python -m pytest -vv --cov=pymc --cov-report=xml --cov-report term --durations=50 $TEST_SUBSET
-      - name: Upload coverage to Codecov
-        uses: codecov/codecov-action@v2
-        with:
-          env_vars: TEST_SUBSET
-          name: ArviZ compat - ${{ matrix.os }} ${{ matrix.floatx }}
-          fail_ci_if_error: false
diff --git a/.github/workflows/jaxtests.yml b/.github/workflows/jaxtests.yml
deleted file mode 100644
index 92f9adc644..0000000000
--- a/.github/workflows/jaxtests.yml
+++ /dev/null
@@ -1,89 +0,0 @@
-name: jax-sampling
-
-on:
-  pull_request:
-    paths:
-      - ".github/workflows/*.yml"
-      - "pymc/**.py"
-      - "*.py"
-      - "conda-envs/**"
-      - "codecov.yml"
-      - "scripts/*.sh"
-  push:
-    branches: [main]
-    paths:
-      - ".github/workflows/*.yml"
-      - "pymc/**.py"
-      - "*.py"
-      - "conda-envs/**"
-      - "codecov.yml"
-      - "scripts/*.sh"
-
-jobs:
-  pytest:
-    strategy:
-      matrix:
-        os: [ubuntu-latest]
-        floatx: [float64]
-        test-subset:
-          - pymc/tests/test_sampling_jax.py
-      fail-fast: false
-    runs-on: ${{ matrix.os }}
-    env:
-      TEST_SUBSET: ${{ matrix.test-subset }}
-      AESARA_FLAGS: floatX=${{ matrix.floatx }},gcc__cxxflags='-march=native'
-    defaults:
-      run:
-        shell: bash -l {0}
-    steps:
-      - uses: actions/checkout@v2
-      - name: Cache conda
-        uses: actions/cache@v1
-        env:
-          # Increase this value to reset cache if environment-test-py39.yml has not changed
-          CACHE_NUMBER: 0
-        with:
-          path: ~/conda_pkgs_dir
-          key: ${{ runner.os }}-conda-${{ env.CACHE_NUMBER }}-${{
-            hashFiles('conda-envs/environment-test-py39.yml') }}
-      - name: Cache multiple paths
-        uses: actions/cache@v2
-        env:
-          # Increase this value to reset cache if requirements.txt has not changed
-          CACHE_NUMBER: 0
-        with:
-          path: |
-            ~/.cache/pip
-            $RUNNER_TOOL_CACHE/Python/*
-            ~\AppData\Local\pip\Cache
-          key: ${{ runner.os }}-build-${{ matrix.python-version }}-${{
-            hashFiles('requirements.txt') }}
-      - uses: conda-incubator/setup-miniconda@v2
-        with:
-          miniforge-variant: Mambaforge
-          miniforge-version: latest
-          mamba-version: "*"
-          activate-environment: pymc-test-py39
-          channel-priority: strict
-          environment-file: conda-envs/environment-test-py39.yml
-          use-mamba: true
-          use-only-tar-bz2: true # IMPORTANT: This needs to be set for caching to work properly!
-      - name: Install pymc
-        run: |
-          conda activate pymc-test-py39
-          pip install -e .
-          python --version
-      - name: Install jax specific dependencies
-        run: |
-          conda activate pymc-test-py39
-          pip install "numpyro>=0.8.0"
-          pip install git+https://github.com/blackjax-devs/blackjax.git@main
-      - name: Run tests
-        run: |
-          python -m pytest -vv --cov=pymc --cov-report=xml --cov-report term --durations=50 $TEST_SUBSET
-      - name: Upload coverage to Codecov
-        uses: codecov/codecov-action@v2
-        with:
-          env_vars: TEST_SUBSET
-          name: JAX tests - ${{ matrix.os }} ${{ matrix.floatx }}
-          fail_ci_if_error: false
diff --git a/.github/workflows/pytest.yml b/.github/workflows/tests.yml
similarity index 63%
rename from .github/workflows/pytest.yml
rename to .github/workflows/tests.yml
index cf73135e29..415562399a 100644
--- a/.github/workflows/pytest.yml
+++ b/.github/workflows/tests.yml
@@ -1,4 +1,4 @@
-name: pytest
+name: tests
 
 on:
   pull_request:
@@ -19,7 +19,6 @@ on:
       - "codecov.yml"
       - "scripts/*.sh"
 
-
 # Tests are split into multiple jobs to accelerate the CI.
 # Different jobs should be organized to take approximately the same
 # time to complete (and not be prohibitely slow).
@@ -249,3 +248,148 @@ jobs:
           env_vars: TEST_SUBSET
           name: ${{ matrix.os }} ${{ matrix.floatx }}
           fail_ci_if_error: false
+  jax:
+    strategy:
+      matrix:
+        os: [ubuntu-latest]
+        floatx: [float64]
+        test-subset:
+          - pymc/tests/test_sampling_jax.py
+      fail-fast: false
+    runs-on: ${{ matrix.os }}
+    env:
+      TEST_SUBSET: ${{ matrix.test-subset }}
+      AESARA_FLAGS: floatX=${{ matrix.floatx }},gcc__cxxflags='-march=native'
+    defaults:
+      run:
+        shell: bash -l {0}
+    steps:
+      - uses: actions/checkout@v2
+      - name: Cache conda
+        uses: actions/cache@v1
+        env:
+          # Increase this value to reset cache if environment-test-py39.yml has not changed
+          CACHE_NUMBER: 0
+        with:
+          path: ~/conda_pkgs_dir
+          key: ${{ runner.os }}-conda-${{ env.CACHE_NUMBER }}-${{
+            hashFiles('conda-envs/environment-test-py39.yml') }}
+      - name: Cache multiple paths
+        uses: actions/cache@v2
+        env:
+          # Increase this value to reset cache if requirements.txt has not changed
+          CACHE_NUMBER: 0
+        with:
+          path: |
+            ~/.cache/pip
+            $RUNNER_TOOL_CACHE/Python/*
+            ~\AppData\Local\pip\Cache
+          key: ${{ runner.os }}-build-${{ matrix.python-version }}-${{
+            hashFiles('requirements.txt') }}
+      - uses: conda-incubator/setup-miniconda@v2
+        with:
+          miniforge-variant: Mambaforge
+          miniforge-version: latest
+          mamba-version: "*"
+          activate-environment: pymc-test-py39
+          channel-priority: strict
+          environment-file: conda-envs/environment-test-py39.yml
+          use-mamba: true
+          use-only-tar-bz2: true # IMPORTANT: This needs to be set for caching to work properly!
+      - name: Install pymc
+        run: |
+          conda activate pymc-test-py39
+          pip install -e .
+          python --version
+      - name: Install jax specific dependencies
+        run: |
+          conda activate pymc-test-py39
+          pip install "numpyro>=0.8.0"
+          pip install git+https://github.com/blackjax-devs/blackjax.git@main
+      - name: Run tests
+        run: |
+          python -m pytest -vv --cov=pymc --cov-report=xml --cov-report term --durations=50 $TEST_SUBSET
+      - name: Upload coverage to Codecov
+        uses: codecov/codecov-action@v2
+        with:
+          env_vars: TEST_SUBSET
+          name: JAX tests - ${{ matrix.os }} ${{ matrix.floatx }}
+          fail_ci_if_error: false
+  arviz:
+    strategy:
+      matrix:
+        os: [ubuntu-latest, macos-latest]
+        floatx: [float64]
+        test-subset:
+          - |
+            pymc/tests/test_parallel_sampling.py
+            pymc/tests/test_posteriors.py
+            pymc/tests/test_sampling.py
+
+          - |
+            pymc/tests/test_data_container.py
+            pymc/tests/test_idata_conversion.py
+            pymc/tests/test_missing.py
+            pymc/tests/test_model.py
+            pymc/tests/test_shape_handling.py
+            pymc/tests/test_shared.py
+      fail-fast: false
+    runs-on: ${{ matrix.os }}
+    env:
+      TEST_SUBSET: ${{ matrix.test-subset }}
+      AESARA_FLAGS: floatX=${{ matrix.floatx }},gcc__cxxflags='-march=native'
+    defaults:
+      run:
+        shell: bash -l {0}
+    steps:
+      - uses: actions/checkout@v2
+      - name: Cache conda
+        uses: actions/cache@v1
+        env:
+          # Increase this value to reset cache if environment-test-py39.yml has not changed
+          CACHE_NUMBER: 0
+        with:
+          path: ~/conda_pkgs_dir
+          key: ${{ runner.os }}-conda-${{ env.CACHE_NUMBER }}-${{
+            hashFiles('conda-envs/environment-test-py39.yml') }}
+      - name: Cache multiple paths
+        uses: actions/cache@v2
+        env:
+          # Increase this value to reset cache if requirements.txt has not changed
+          CACHE_NUMBER: 0
+        with:
+          path: |
+            ~/.cache/pip
+            $RUNNER_TOOL_CACHE/Python/*
+            ~\AppData\Local\pip\Cache
+          key: ${{ runner.os }}-build-${{ matrix.python-version }}-${{
+            hashFiles('requirements.txt') }}
+      - uses: conda-incubator/setup-miniconda@v2
+        with:
+          miniforge-variant: Mambaforge
+          miniforge-version: latest
+          mamba-version: "*"
+          activate-environment: pymc-test-py39
+          channel-priority: strict
+          environment-file: conda-envs/environment-test-py39.yml
+          use-mamba: true
+          use-only-tar-bz2: true # IMPORTANT: This needs to be set for caching to work properly!
+      - name: Install pymc
+        run: |
+          conda activate pymc-test-py39
+          pip install -e .
+          python --version
+      - name: Install latest arviz
+        run: |
+          conda activate pymc-test-py39
+          pip uninstall arviz -y
+          pip install git+https://github.com/arviz-devs/arviz
+      - name: Run tests
+        run: |
+          python -m pytest -vv --cov=pymc --cov-report=xml --cov-report term --durations=50 $TEST_SUBSET
+      - name: Upload coverage to Codecov
+        uses: codecov/codecov-action@v2
+        with:
+          env_vars: TEST_SUBSET
+          name: ArviZ compat - ${{ matrix.os }} ${{ matrix.floatx }}
+          fail_ci_if_error: false
diff --git a/buildosx b/buildosx
deleted file mode 100755
index 3fff5a694a..0000000000
--- a/buildosx
+++ /dev/null
@@ -1,6 +0,0 @@
-#!/bin/sh
-export CFLAGS="-arch x86_64"
-export FFLAGS="-static -ff2c -arch x86_64"
-export LDFLAGS="-Wall -undefined dynamic_lookup -bundle -arch x86_64"
-rm -rf build
-python setupegg.py bdist_egg
diff --git a/scripts/check_all_tests_are_covered.py b/scripts/check_all_tests_are_covered.py
index 6c4b40290d..6d83a1e83c 100644
--- a/scripts/check_all_tests_are_covered.py
+++ b/scripts/check_all_tests_are_covered.py
@@ -36,7 +36,7 @@ def from_yaml():
     """
     # First collect the matrix definitions from testing workflows
     matrices = {}
-    for wf in ["pytest.yml", "arviz_compat.yml", "jaxtests.yml"]:
+    for wf in ["tests.yml"]:
         wfname = wf.strip(".yml")
         wfdef = yaml.safe_load(open(Path(".github", "workflows", wf)))
         for jobname, jobdef in wfdef["jobs"].items():


Code review comments:
diff_hunk: @@ -249,3 +248,148 @@ jobs:
           env_vars: TEST_SUBSET
           name: ${{ matrix.os }} ${{ matrix.floatx }}
           fail_ci_if_error: false
+  jax:
+    strategy:
+      matrix:
+        os: [ubuntu-latest]
+        floatx: [float64]
+        test-subset:
+          - pymc/tests/test_sampling_jax.py
+      fail-fast: false
+    runs-on: ${{ matrix.os }}
+    env:
+      TEST_SUBSET: ${{ matrix.test-subset }}
+      AESARA_FLAGS: floatX=${{ matrix.floatx }},gcc__cxxflags='-march=native'
+    defaults:
+      run:
+        shell: bash -l {0}
+    steps:
+      - uses: actions/checkout@v2
+      - name: Cache conda
+        uses: actions/cache@v1
+        env:
+          # Increase this value to reset cache if environment-test-py39.yml has not changed
+          CACHE_NUMBER: 0
+        with:
+          path: ~/conda_pkgs_dir
+          key: ${{ runner.os }}-conda-${{ env.CACHE_NUMBER }}-${{
+            hashFiles('conda-envs/environment-test-py39.yml') }}
+      - name: Cache multiple paths
+        uses: actions/cache@v2
+        env:
+          # Increase this value to reset cache if requirements.txt has not changed
+          CACHE_NUMBER: 0
+        with:
+          path: |
+            ~/.cache/pip
+            $RUNNER_TOOL_CACHE/Python/*
+            ~\AppData\Local\pip\Cache
+          key: ${{ runner.os }}-build-${{ matrix.python-version }}-${{
+            hashFiles('requirements.txt') }}
+      - uses: conda-incubator/setup-miniconda@v2
+        with:
+          miniforge-variant: Mambaforge
+          miniforge-version: latest
+          mamba-version: "*"
+          activate-environment: pymc-test-py39
+          channel-priority: strict
+          environment-file: conda-envs/environment-test-py39.yml
+          use-mamba: true
+          use-only-tar-bz2: true # IMPORTANT: This needs to be set for caching to work properly!
+      - name: Install pymc
+        run: |
+          conda activate pymc-test-py39
+          pip install -e .
+          python --version
+      - name: Install jax specific dependencies
+        run: |
+          conda activate pymc-test-py39
+          pip install "numpyro>=0.8.0"
+          pip install git+https://github.com/blackjax-devs/blackjax.git@main
+      - name: Run tests
+        run: |
+          python -m pytest -vv --cov=pymc --cov-report=xml --cov-report term --durations=50 $TEST_SUBSET
+      - name: Upload coverage to Codecov
+        uses: codecov/codecov-action@v2
+        with:
+          env_vars: TEST_SUBSET
+          name: JAX tests - ${{ matrix.os }} ${{ matrix.floatx }}
+          fail_ci_if_error: false
+  arviz:
path: .github/workflows/tests.yml
text: @user1:
Can we give this a better name? Perhaps "latest" or "latest_dependency"? It's not that much focused on arviz (at least not anymore)

@user1:
Or perhaps just get rid of it and just have a windows-macos job?