Repository: dask/dask

Issue:
Make yaml a required dependency
Currently `dask/dask` doesn't depend directly on `yaml`, even though the config system uses it. This is because historically we wanted a minimal install (`pip install dask`) of just the core scheduler and graph routines (including optimization) to work with no dependencies.

This limitation means that in `dask/dask`, all `dask.config.get` calls need to also provide a default, since the default `dask.yaml` file may not be read in at all.

In practice this makes developing `dask/dask` using the config system complicated - the default values need to be stored in both the code and the config file. Further, the only test we have for this behavior (a minimal install that *doesn't* include yaml) runs on a cron job, no part of PRs. It also doesn't hit all cases - we've had long existing code paths that would break if `yaml` wasn't also present (e.g. https://github.com/dask/dask/blob/master/dask/array/core.py#L2543).

I'd like to propose that we make `yaml` a required dependency for `dask/dask`. This would make the minimal install no longer dependency-less, but I think at this point dependencies in the Python ecosystem are no longer as tricky to install as they were 5 years ago. At minimum, I think it should be required for all the collections (e.g `dask.array`), as that reduces the number of files where we need to test this behavior.


Discussion:
This is causing some current test failures, so I'd like to reach a resolution mildly quickly. cc @mrocklin, @TomAugspurger, @jrbourbeau for thoughts.
Initial reaction: adding it for the array / dataframe / bag extras makes
complete sense.

On Mon, May 18, 2020 at 8:59 AM Jim Crist-Harif <notifications@github.com>
wrote:

> This is causing some current test failures, so I'd like to reach a
> resolution mildly quickly. cc @mrocklin <https://github.com/mrocklin>,
> @TomAugspurger <https://github.com/TomAugspurger>, @jrbourbeau
> <https://github.com/jrbourbeau> for thoughts.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/dask/issues/6221#issuecomment-630201576>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAKAOIXA2BTLPHPH2PUVUCLRSE5MJANCNFSM4NEDJIQA>
> .
>

I agree with @TomAugspurger 's reaction that having it for anything on top of core Dask makes sense.  

I don't have strong opinions on adding it as a core dependency.  My gut reaction for and against are "I'm not sure that anyone cares about super-lightweight dependencies any longer" and "it's decently easy to add defaults in the core parts where we can't assume yaml" although presumably this came about because the latter gut reaction failed.
Since we don't have a test for it that runs on PRs, it's easy for these to slip through. I think since we're already struggling for developer bandwidth, making things harder by having two codepaths (one with yaml, one without) isn't worth it. If a user is using the distributed scheduler (many of our users) they'll already have it installed.
For what it's worth, I'd be strongly in favour. As a rule of thumb, I always prefer to make all small and pure-python dependencies mandatory in order to reduce the number of degrees of freedom that need to be covered by testing. Don't ship what you don't test.
fwiw again, the lack of pyyaml dependency broke CI for both napari and scikit-image with 2.17 with the "KeyError: 'optimization'" error first reported in https://github.com/dask/dask/pull/6198#issuecomment-629993387

Looking at the references piling up there we are not exceptions! :grimacing: 
I don't find any of the arguments for keeping `pip install dask` dependency free compelling enough to not make yaml a core dependency here. Since most users will also have `distributed` installed (which requires `yaml`), and it's hard for us as developers to ensure this (we don't currently test for it on PRs), and it's a restriction that's easy to forget, I think we should do the easy thing and just make it a dependency.

I'll push a PR up now.

Resolving patch:
diff --git a/dask/config.py b/dask/config.py
--- a/dask/config.py
+++ b/dask/config.py
@@ -6,10 +6,7 @@
 import threading
 import warnings
 
-try:
-    import yaml
-except ImportError:
-    yaml = None
+import yaml
 
 
 no_default = "__no_default__"
@@ -387,11 +384,8 @@ def collect(paths=paths, env=None):
     """
     if env is None:
         env = os.environ
-    configs = []
-
-    if yaml:
-        configs.extend(collect_yaml(paths=paths))
 
+    configs = collect_yaml(paths=paths)
     configs.append(collect_env(env=env))
 
     return merge(*configs)
@@ -581,10 +575,7 @@ def check_deprecations(key: str, deprecations: dict = deprecations):
         return key
 
 
-refresh()
-
-
-if yaml:
+def _initialize():
     fn = os.path.join(os.path.dirname(__file__), "dask.yaml")
     ensure_file(source=fn)
 
@@ -592,4 +583,7 @@ def check_deprecations(key: str, deprecations: dict = deprecations):
         _defaults = yaml.safe_load(f)
 
     update_defaults(_defaults)
-    del fn, _defaults
+
+
+refresh()
+_initialize()
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -26,9 +26,8 @@
     "diagnostics": ["bokeh >= 1.0.0"],
     "delayed": ["cloudpickle >= 0.2.2", "toolz >= 0.8.2"],
 }
-extras_require["complete"] = sorted(
-    {"PyYaml"} | {v for req in extras_require.values() for v in req}
-)
+
+install_requires = ["pyyaml"]
 
 packages = [
     "dask",
@@ -68,7 +67,7 @@
     packages=packages + tests,
     long_description=open("README.rst").read() if exists("README.rst") else "",
     python_requires=">=3.6",
-    install_requires=[],
+    install_requires=install_requires,
     setup_requires=setup_requires,
     tests_require=["pytest"],
     extras_require=extras_require,


Test patch:
diff --git a/continuous_integration/travis/test_imports.sh b/continuous_integration/travis/test_imports.sh
--- a/continuous_integration/travis/test_imports.sh
+++ b/continuous_integration/travis/test_imports.sh
@@ -5,7 +5,7 @@ set -o errexit
 test_import () {
     echo "Create environment: python=$PYTHON $1"
     # Create an empty environment
-    conda create -y -n test-imports -c conda-forge python=$PYTHON_VERSION $1 > /dev/null 2>&1
+    conda create -y -n test-imports -c conda-forge python=$PYTHON_VERSION pyyaml $1 > /dev/null 2>&1
     source activate test-imports > /dev/null 2>&1
     echo "python -c '$2'"
     python -c "$2"