Repository: dask/distributed

Issue:
Computing delayed None fails
I regularly find myself delaying data. I encountered an issue when delaying None: it works fine with simple dask schedulers like processes, but fails with distributed, at least with a local cluster:

```
from dask import delayed
from dask.distributed import Client, LocalCluster

client = Client(LocalCluster())

x = delayed(12)
x.compute(scheduler="processes")  # Returns: 12
x.compute(scheduler=client)  # Returns: 12

y = delayed(None)
y.compute(scheduler="processes")  # Returns: None
y.compute(scheduler=client)  # Error: TypeError: 'NoneType' object is not callable
```

Passing the delayed value as an input to a delayed function yields the same behavior:

```
delayed(str)(y).compute(scheduler="processes")  # Returns: 'None'
delayed(str)(y).compute(scheduler=client)  # Error: TypeError: 'NoneType' object is not callable
```

For now, the workaround I use for the purpose of my project is to detect and replace None values with a delayed function that returns None:

```
def none_func():
    return None

z = delayed(none_func)()
z.compute(scheduler="processes")  # Returns: None
z.compute(scheduler=client)  # Returns: None
```

But I would be happy to switch back to simpler code.


Discussion:
I'm not totally sure what is happening here but `yield None`, which eventually happens down in the tornado world, is [special](http://www.tornadoweb.org/en/stable/gen.html#tornado.gen.moment).
Interesting.  Thanks for the excellent reproducible example @alex-marty .  I'll take a look later today.

> I'm not totally sure what is happening here but yield None, which eventually happens down in the tornado world, is special.

Also interesting.  I'll keep that in mind as I play around.

Resolving patch:
diff --git a/distributed/worker.py b/distributed/worker.py
--- a/distributed/worker.py
+++ b/distributed/worker.py
@@ -1216,7 +1216,7 @@ def add_task(
         function=None,
         args=None,
         kwargs=None,
-        task=None,
+        task=no_value,
         who_has=None,
         nbytes=None,
         priority=None,
@@ -3007,7 +3007,7 @@ def get_data_from_worker(
 job_counter = [0]
 
 
-def _deserialize(function=None, args=None, kwargs=None, task=None):
+def _deserialize(function=None, args=None, kwargs=None, task=no_value):
     """ Deserialize task inputs and regularize to func, args, kwargs """
     if function is not None:
         function = pickle.loads(function)
@@ -3016,7 +3016,7 @@ def _deserialize(function=None, args=None, kwargs=None, task=None):
     if kwargs:
         kwargs = pickle.loads(kwargs)
 
-    if task is not None:
+    if task is not no_value:
         assert not function and not args and not kwargs
         function = execute_task
         args = (task,)


Test patch:
diff --git a/distributed/tests/test_collections.py b/distributed/tests/test_collections.py
--- a/distributed/tests/test_collections.py
+++ b/distributed/tests/test_collections.py
@@ -6,6 +6,7 @@
 pytest.importorskip("numpy")
 pytest.importorskip("pandas")
 
+import dask
 import dask.dataframe as dd
 import dask.bag as db
 from distributed.client import wait
@@ -185,3 +186,12 @@ def test_sparse_arrays(c, s, a, b):
     future = c.compute(s.sum(axis=0)[:10])
 
     yield future
+
+
+@gen_cluster(client=True, ncores=[("127.0.0.1", 1)])
+def test_delayed_none(c, s, w):
+    x = dask.delayed(None)
+    y = dask.delayed(123)
+    [xx, yy] = c.compute([x, y])
+    assert (yield xx) is None
+    assert (yield yy) == 123