Repository: dask/dask

Issue:
cloudpickle requirement
``multiprocessing`` imports  ``cloudpickle`` since deb74696e859c. Shouldn't  ``cloudpickle`` be in ``setup.py``'s ``install_req``?


Discussion:
`multiprocessing` imported cloudpickle before then (scroll down to https://github.com/dask/dask/commit/deb74696e859cd50fcb2271a810811364aa27a59#diff-5edbcf144004ac2002f1bf6310a31500L22 and you can see the import was just moved). 

Cloudpickle is in the `setup.py`, but in the `extras_require`. Dask is broken into small components - the core (really just the spec) has no dependencies, but each of the extras do require things. To install everything, do `pip install dask[complete]`. This allows subprojects to use dask, but not require everything (e.g. numpy). 
The thing is that cloudpickle is in ``extra_requires`` for ``bag`` and ``dataframe``, but we are not using any of them and still dask seems to require cloudpickle. With dask 0.13 we didn't need to install cloudpickle manually.
Oh, I see what the issue is. The multiprocessing scheduler has required cloudpickle for a long time (well before `0.13.0`), but cloudpickle has never been (and shouldn't be) a requirement for dask core. The issue is the addition of [this import](https://github.com/dask/dask/blob/master/dask/utils.py#L20) in https://github.com/dask/dask/pull/1919. Will fix, and add a test that importing with no dependencies works.
Great, thanks!

Resolving patch:
diff --git a/dask/utils.py b/dask/utils.py
--- a/dask/utils.py
+++ b/dask/utils.py
@@ -17,7 +17,6 @@
 from importlib import import_module
 from threading import Lock
 import multiprocessing as mp
-from . import multiprocessing
 import uuid
 from weakref import WeakValueDictionary
 
@@ -1033,6 +1032,7 @@ def effective_get(get=None, collection=None):
 def get_scheduler_lock(get=None, collection=None):
     """Get an instance of the appropriate lock for a certain situation based on
        scheduler used."""
+    from . import multiprocessing
     actual_get = effective_get(get, collection)
 
     if actual_get == multiprocessing.get:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -4,13 +4,15 @@
 from setuptools import setup
 import versioneer
 
+# NOTE: These are tested in `continuous_integration/travis/test_imports.sh` If
+# you modify these, make sure to change the corresponding line there.
 extras_require = {
   'array': ['numpy', 'toolz >= 0.7.2'],
   'bag': ['cloudpickle >= 0.2.1', 'toolz >= 0.7.2', 'partd >= 0.3.7'],
   'dataframe': ['numpy', 'pandas >= 0.19.0', 'toolz >= 0.7.2',
                 'partd >= 0.3.7', 'cloudpickle >= 0.2.1'],
   'distributed': ['distributed >= 1.15', 's3fs >= 0.0.8'],
-  'imperative': ['toolz >= 0.7.2'],
+  'delayed': ['toolz >= 0.7.2'],
 }
 extras_require['complete'] = sorted(set(sum(extras_require.values(), [])))
 


Test patch:
diff --git a/continuous_integration/run_tests.cmd b/continuous_integration/appveyor/run_tests.cmd
similarity index 100%
rename from continuous_integration/run_tests.cmd
rename to continuous_integration/appveyor/run_tests.cmd
diff --git a/continuous_integration/travis/run_tests.sh b/continuous_integration/travis/run_tests.sh
new file mode 100644
--- /dev/null
+++ b/continuous_integration/travis/run_tests.sh
@@ -0,0 +1,12 @@
+# Need to make test order deterministic when parallelizing tests, hence PYTHONHASHSEED
+# (see https://github.com/pytest-dev/pytest-xdist/issues/63)
+if [[ $PARALLEL == 'true' ]]; then
+    export XTRATESTARGS="-n3 $XTRATESTARGS"
+    export PYTHONHASHSEED=42
+fi
+
+if [[ $COVERAGE == 'true' ]]; then
+    coverage run `which py.test` dask --runslow --doctest-modules --verbose $XTRATESTARGS
+else
+    py.test dask --runslow --verbose $XTRATESTARGS
+fi
diff --git a/continuous_integration/travis/test_imports.sh b/continuous_integration/travis/test_imports.sh
new file mode 100644
--- /dev/null
+++ b/continuous_integration/travis/test_imports.sh
@@ -0,0 +1,34 @@
+test_import () {
+    # Install dependencies
+    if [[ -n "$2" ]]; then
+        output=$(conda install -c conda-forge $2)
+        if [[ $? -eq 1 ]]; then
+            echo $output
+            echo "$1 install failed" >&2
+            exit 1
+        fi
+    fi
+    # Check import
+    python -c "$3"
+    if [[ $? -eq 1 ]]; then
+        echo "$1 import failed" >&2
+        exit 1
+    else
+        echo "$1 import succeeded"
+    fi
+    # Uninstall dependencies
+    if [[ -n "$2" ]]; then
+        output=$(conda uninstall $2)
+    fi
+}
+
+# Create an empty environment
+conda create -n test-imports python=$PYTHON
+source activate test-imports
+
+(test_import "Core" "" "import dask, dask.threaded, dask.optimize") && \
+(test_import "Delayed" "toolz" "import dask.delayed") && \
+(test_import "Bag" "toolz partd cloudpickle" "import dask.bag") && \
+(test_import "Array" "toolz numpy" "import dask.array") && \
+(test_import "Dataframe" "numpy pandas toolz partd cloudpickle" "import dask.dataframe") && \
+(test_import "Distributed" "distributed s3fs" "import dask.distributed")