Repository: dask/dask
Pull request #8275 (2021-10-19 17:37:37)

Get rid of environment-latest and update to python 3.9
This is an alternative to #8259. The failures in upstream tests match those on main.

Closes https://github.com/dask/dask/issues/6808

raybellwaves: > @raybellwaves I think you added environment-latest, so just wanted to get your thoughts on this.

Go for it.

Patch:
diff --git a/.github/workflows/additional.yml b/.github/workflows/additional.yml
index d3c0554b7..13dea056e 100644
--- a/.github/workflows/additional.yml
+++ b/.github/workflows/additional.yml
@@ -90,8 +90,8 @@ jobs:
           miniforge-version: latest
           use-mamba: true
           channel-priority: strict
-          python-version: "3.8"
-          environment-file: continuous_integration/environment-latest.yaml
+          python-version: "3.9"
+          environment-file: continuous_integration/environment-3.9.yaml
           activate-environment: test-environment
           auto-activate-base: false
 
diff --git a/.github/workflows/upstream.yml b/.github/workflows/upstream.yml
index 54af0b96d..124f80dac 100644
--- a/.github/workflows/upstream.yml
+++ b/.github/workflows/upstream.yml
@@ -46,8 +46,8 @@ jobs:
           miniforge-version: latest
           use-mamba: true
           channel-priority: strict
-          python-version: "3.8"
-          environment-file: continuous_integration/environment-3.8-dev.yaml
+          python-version: "3.9"
+          environment-file: continuous_integration/environment-3.9.yaml
           activate-environment: test-environment
           auto-activate-base: false
 
diff --git a/continuous_integration/environment-3.8-dev.yaml b/continuous_integration/environment-3.8-dev.yaml
deleted file mode 100644
index 4c6787b9f..000000000
--- a/continuous_integration/environment-3.8-dev.yaml
+++ /dev/null
@@ -1,64 +0,0 @@
-# This job tests against upstream dev
-name: test-environment
-channels:
-  - conda-forge
-dependencies:
-  # required dependencies
-  - python=3.8
-  - packaging
-  # For the dependencies. These will be re-installed later.
-  - numpy
-  - pandas
-  # test dependencies
-  - pre-commit
-  - pytest
-  - pytest-cov
-  - pytest-rerunfailures
-  - pytest-xdist
-  - moto
-  - flask
-  - fastparquet
-  - h5py
-  - pytables
-  - zarr
-  - tiledb-py
-  - xarray
-  # sqlalchemy 1.4.0 causes deprecation warnings to be raised from pandas
-  # along with other issues https://github.com/pandas-dev/pandas/issues/40467
-  - sqlalchemy<1.4.0
-  - pyarrow
-  - jsonschema
-  # other -- IO
-  - bcolz
-  - blosc
-  # Need recent version of s3fs to support newer aiobotocore versions
-  # https://github.com/dask/s3fs/issues/514
-  - s3fs>=2021.8.0
-  - fsspec
-  - boto3
-  - botocore
-  - aiohttp
-  - bokeh
-  - httpretty
-  - cloudpickle
-  - crick
-  - cytoolz
-  - distributed
-  - graphviz
-  - ipython
-  - lz4
-  - numba
-  - partd
-  - psutil
-  - requests
-  - scikit-image
-  - scikit-learn
-  - scipy
-  - toolz
-  - python-snappy
-  - cachey
-  - python-graphviz
-  - python-xxhash
-  - mmh3
-  - jinja2
-  - pip
diff --git a/continuous_integration/environment-3.9.yaml b/continuous_integration/environment-3.9.yaml
index c6a3a2ad6..f8771e8be 100644
--- a/continuous_integration/environment-3.9.yaml
+++ b/continuous_integration/environment-3.9.yaml
@@ -30,7 +30,7 @@ dependencies:
   - coverage
   - jsonschema
   # other -- IO
-  # Not available for Python 3.9 on conda-forge yet
+  # Not available for Python 3.9 on conda-forge
   # - bcolz
   - blosc
   - boto3
diff --git a/docs/source/develop.rst b/docs/source/develop.rst
index 6721499aa..698af706a 100644
--- a/docs/source/develop.rst
+++ b/docs/source/develop.rst
@@ -105,7 +105,7 @@ pip or conda_
 
 ``conda``::
 
-  conda env create -n dask-dev -f continuous_integration/environment-latest.yaml
+  conda env create -n dask-dev -f continuous_integration/environment-3.9.yaml
   conda activate dask-dev
   python -m pip install --no-deps -e .
 


Test patch:
diff --git a/continuous_integration/environment-latest.yaml b/continuous_integration/environment-latest.yaml
deleted file mode 120000
index f191f6635..000000000
--- a/continuous_integration/environment-latest.yaml
+++ /dev/null
@@ -1,1 +0,0 @@
-environment-3.8.yaml
\ No newline at end of file
diff --git a/dask/tests/test_docs.py b/dask/tests/test_docs.py
new file mode 100644
index 000000000..4bd196838
--- /dev/null
+++ b/dask/tests/test_docs.py
@@ -0,0 +1,17 @@
+from pathlib import Path
+
+
+def test_development_guidelines_matches_ci():
+    """When the environment.yaml changes in CI, make sure to change it in the docs as well"""
+    root_dir = Path(__file__).parent.parent.parent
+
+    development_doc_file = root_dir / "docs" / "source" / "develop.rst"
+    additional_ci_file = root_dir / ".github" / "workflows" / "additional.yml"
+    upstream_ci_file = root_dir / ".github" / "workflows" / "upstream.yml"
+    latest_env = "environment-3.9.yaml"
+
+    for filename in [development_doc_file, additional_ci_file, upstream_ci_file]:
+        with open(filename, encoding="utf8") as f:
+            assert any(
+                latest_env in line for line in f
+            ), f"{latest_env} not found in {filename}"