Repository: dask/distributed

Issue:
Scattering data to workers when using Queues
Hello everyone, 

to my understanding it should be possible to distribute data to the worker nodes using the `client.scatter` method. @mrocklin described it in this stackoverflow post [here](http://stackoverflow.com/questions/41471248/how-to-efficiently-submit-tasks-with-large-arguments-in-dask-distributed). And indeed that seems to be working fine when batch processing data. When using queues (similar to #838 ) it does not seem to work. Heres a small example:

```
from queue import Queue
import time
from distributed import Client
from threading import Thread

client = Client('127.0.0.1:8786')


def load_data(q):
    i = 0
    while True:
        q.put(i)
        time.sleep(0.1)
        i += 1


def increment(x, inc=[0, 0, 0]):
    time.sleep(0.5)
    for i in inc:
        x += i
    return x


input_q = Queue(maxsize=100)
data = [1, 2, 3, 4, 5, 6, 7, 8]

remote_data = client.scatter(data)
remote_q = client.scatter(input_q)


inc_q = client.map(increment, remote_q, **{'inc': remote_data})
result_q = client.gather(inc_q)


Thread(target=load_data, args=(input_q,), daemon=True).start()

while True:
    print('latest result: {}'.format(result_q.get()))

```

The dask-scheduler produces some  errors.

```
distributed.scheduler - INFO - Receive client connection: 7e7c6e14-e7ba-11e6-8335-985aebd351e4
distributed.scheduler - ERROR - 'increment-72c55d32d351ec069e5d00cd2bbf4f1c'
Traceback (most recent call last):
  File "/Users/kbruegge/Development/distributed/distributed/scheduler.py", line 2182, in transition_processing_memory
    s.remove(key)
KeyError: 'increment-72c55d32d351ec069e5d00cd2bbf4f1c'
distributed.scheduler - ERROR - 'increment-72c55d32d351ec069e5d00cd2bbf4f1c'
Traceback (most recent call last):
  File "/Users/kbruegge/Development/distributed/distributed/scheduler.py", line 2655, in transition
    recommendations = func(key, *args, **kwargs)
  File "/Users/kbruegge/Development/distributed/distributed/scheduler.py", line 2182, in transition_processing_memory
    s.remove(key)
KeyError: 'increment-72c55d32d351ec069e5d00cd2bbf4f1c'
distributed.scheduler - ERROR - 'increment-72c55d32d351ec069e5d00cd2bbf4f1c'
Traceback (most recent call last):
  File "/Users/kbruegge/Development/distributed/distributed/scheduler.py", line 1249, in handle_worker
    handler(worker=worker, **msg)
  File "/Users/kbruegge/Development/distributed/distributed/scheduler.py", line 1191, in handle_task_finished
    r = self.stimulus_task_finished(key=key, **msg)
  File "/Users/kbruegge/Development/distributed/distributed/scheduler.py", line 719, in stimulus_task_finished
    **kwargs)
  File "/Users/kbruegge/Development/distributed/distributed/scheduler.py", line 2655, in transition
    recommendations = func(key, *args, **kwargs)
  File "/Users/kbruegge/Development/distributed/distributed/scheduler.py", line 2182, in transition_processing_memory
    s.remove(key)
KeyError: 'increment-72c55d32d351ec069e5d00cd2bbf4f1c'
distributed.scheduler - INFO - Remove worker tcp://127.0.0.1:50183
distributed.scheduler - INFO - Plugin failed with exception
Traceback (most recent call last):
  File "/Users/kbruegge/Development/distributed/distributed/scheduler.py", line 1249, in handle_worker
    handler(worker=worker, **msg)
  File "/Users/kbruegge/Development/distributed/distributed/scheduler.py", line 1191, in handle_task_finished
    r = self.stimulus_task_finished(key=key, **msg)
  File "/Users/kbruegge/Development/distributed/distributed/scheduler.py", line 719, in stimulus_task_finished
    **kwargs)
  File "/Users/kbruegge/Development/distributed/distributed/scheduler.py", line 2655, in transition
    recommendations = func(key, *args, **kwargs)
  File "/Users/kbruegge/Development/distributed/distributed/scheduler.py", line 2182, in transition_processing_memory
    s.remove(key)
KeyError: 'increment-72c55d32d351ec069e5d00cd2bbf4f1c'

During handling of the above exception, another exception occurred:
.
.
.

```

It goes on like this for a while. Then some lines of 

```
distributed.scheduler - INFO - User asked for computation on lost data, increment-385d81d312ae7cfbb71b30ba3249906a
distributed.scheduler - INFO - User asked for computation on lost data, increment-74ceedfca5f3ec3db206b0eeeb68bfb1
distributed.scheduler - INFO - User asked for computation on lost data, increment-7fd9b944879e6900fb10ed810f222067
distributed.scheduler - INFO - User asked for computation on lost data, increment-8f151ea5798ad438e86823b19652200d
.
.
.
```

And then more stacktraces.  
It does work fine when batch processing data. Is there another approach I should take to distribute shared objects to the workers?




Discussion:
I'm doing this on the latest master branch by the way. Commit 644e97dd555783caa204a5b2fb2fb8c4b610ee4c to be precise. 

Resolving patch:
diff --git a/distributed/scheduler.py b/distributed/scheduler.py
--- a/distributed/scheduler.py
+++ b/distributed/scheduler.py
@@ -932,6 +932,7 @@ def validate_waiting(self, key):
         assert key not in self.released
         for dep in self.dependencies[key]:
             assert (dep in self.who_has) + (dep in self.waiting[key]) == 1
+            assert key in self.waiting_data[dep]
 
     def validate_processing(self, key):
         assert key not in self.waiting
@@ -942,6 +943,7 @@ def validate_processing(self, key):
         assert key not in self.who_has
         for dep in self.dependencies[key]:
             assert dep in self.who_has
+            assert key in self.waiting_data[dep]
 
     def validate_memory(self, key):
         assert key in self.who_has
@@ -1722,7 +1724,8 @@ def update_data(self, comm=None, who_has=None, nbytes=None, client=None):
                                                             DEFAULT_DATA_SIZE)
                     self.has_what[w].add(key)
                     self.who_has[key].add(w)
-                self.waiting_data[key] = set()
+                if key not in self.waiting_data:
+                    self.waiting_data[key] = set()
                 self.report({'op': 'key-in-memory',
                              'key': key,
                              'workers': list(workers)})


Test patch:
diff --git a/distributed/tests/test_client.py b/distributed/tests/test_client.py
--- a/distributed/tests/test_client.py
+++ b/distributed/tests/test_client.py
@@ -106,6 +106,7 @@ def f(x, y=10):
     s.validate_state()
 
 
+
 @gen_cluster(client=True)
 def test_map_empty(c, s, a, b):
     L1 = c.map(inc, [], pure=False)
@@ -3690,3 +3691,37 @@ def test_client_timeout():
 
     yield c._shutdown()
     yield s.close()
+
+
+@gen_cluster(client=True)
+def test_submit_list_kwargs(c, s, a, b):
+    futures = yield c._scatter([1, 2, 3])
+    def f(L=None):
+        return sum(L)
+
+    future = c.submit(f, L=futures)
+    result = yield future._result()
+    assert result == 1 + 2 + 3
+
+
+@gen_cluster(client=True)
+def test_map_list_kwargs(c, s, a, b):
+    futures = yield c._scatter([1, 2, 3])
+    def f(i, L=None):
+        return i + sum(L)
+
+    futures = c.map(f, range(10), L=futures)
+    results = yield c._gather(futures)
+    assert results == [i + 6 for i in range(10)]
+
+
+@gen_cluster(client=True)
+def test_dont_clear_waiting_data(c, s, a, b):
+    [x] = yield c._scatter([1])
+    y = c.submit(slowinc, x, delay=0.2)
+    while y.key not in s.task_state:
+        yield gen.sleep(0.01)
+    [x] = yield c._scatter([1])
+    for i in range(5):
+        assert s.waiting_data[x.key]
+        yield gen.moment