internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: dask/distributed

Issue:
Random serialization bug in scheduler's handle_messages
When working with a large number of dask.delay'd tasks (>= 5e3) and a large number of workers (3 procs x 32 cores or  5 procs x 32 cores) I sometimes (but quite rarely) get the following low level error:

```
Traceback (most recent call last):
  File "/work/miniconda/lib/python3.5/site-packages/distributed/scheduler.py", line 1268, in handle_messages
    msgs = yield next_message()  # in_queue.get()
  File "/work/miniconda/lib/python3.5/site-packages/tornado/gen.py", line 1008, in run
    value = future.result()
  File "/work/miniconda/lib/python3.5/site-packages/tornado/concurrent.py", line 232, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 3, in raise_exc_info
  File "/work/miniconda/lib/python3.5/site-packages/tornado/gen.py", line 1017, in run
    yielded = self.gen.send(value)
  File "/work/miniconda/lib/python3.5/site-packages/distributed/core.py", line 255, in read
    msg = protocol.loads(frames)
  File "/work/miniconda/lib/python3.5/site-packages/distributed/protocol.py", line 91, in loads
    msg = loads_msgpack(header, payload)
  File "/work/miniconda/lib/python3.5/site-packages/distributed/protocol.py", line 186, in loads_msgpack
    return msgpack.loads(payload, encoding='utf8')
  File "pandas/msgpack/_unpacker.pyx", line 142, in pandas.msgpack._unpacker.unpackb (pandas/msgpack/_unpacker.cpp:2134)
pandas.msgpack.exceptions.ExtraData: unpack(b) received extra data.
```

Once it happens the executor is stuck (with no additional errors): all connections seems to block indefinitely and I have to recreate a new executor instance to be able to resume work.

I could not manage to reproduce this on a single host with exactly the same code:

https://github.com/ogrisel/docker-distributed/blob/master/examples/sklearn_parameter_search.ipynb



Discussion:

Resolving patch:
diff --git a/distributed/core.py b/distributed/core.py
--- a/distributed/core.py
+++ b/distributed/core.py
@@ -270,10 +270,12 @@ def write(stream, msg):
 
         lengths = ([struct.pack('Q', len(frames))] +
                    [struct.pack('Q', len(frame)) for frame in frames])
-        yield stream.write(b''.join(lengths))
+        stream.write(b''.join(lengths))
 
-        for frame in frames:
-            yield stream.write(frame)
+        for frame in frames[:-1]:
+            stream.write(frame)
+
+        yield stream.write(frames[-1])
 
 
 def pingpong(stream):


Test patch:
Repository: dask/distributed

Issue:
LZ4 compression fails on very large frames
This causes a complete halt of the system.  We could consider framing or punting.



Discussion:

Resolving patch:
diff --git a/distributed/protocol.py b/distributed/protocol.py
--- a/distributed/protocol.py
+++ b/distributed/protocol.py
@@ -198,6 +198,17 @@ def dumps_big_byte_dict(d):
         loads_big_byte_dict
     """
     assert isinstance(d, dict) and all(isinstance(v, bytes) for v in d.values())
+    shards = {}
+    for k, v in list(d.items()):
+        if len(v) >= 2**31:
+            L = []
+            for i, j in enumerate(range(0, len(v), 2**30)):
+                key = '.shard-%d-%s' % (i, k)
+                d[key] = v[j: j + 2**30]
+                L.append(key)
+            del d[k]
+            shards[k] = L
+
     keys, values = zip(*d.items())
 
     compress = compressions[default_compression]['compress']
@@ -211,6 +222,8 @@ def dumps_big_byte_dict(d):
     header = {'encoding': 'big-byte-dict',
               'keys': keys,
               'compression': compression}
+    if shards:
+        header['shards'] = shards
 
     return [msgpack.dumps(header, use_bin_type=True)] + values2
 
@@ -225,4 +238,8 @@ def loads_big_byte_dict(header, *values):
 
     values2 = [compressions[c]['decompress'](v)
                for c, v in zip(header['compression'], values)]
-    return dict(zip(header['keys'], values2))
+    result = dict(zip(header['keys'], values2))
+
+    for k, keys in header.get('shards', {}).items():
+        result[k] = b''.join(result.pop(kk) for kk in keys)
+    return result


Test patch:
diff --git a/distributed/tests/test_protocol.py b/distributed/tests/test_protocol.py
--- a/distributed/tests/test_protocol.py
+++ b/distributed/tests/test_protocol.py
@@ -79,3 +79,15 @@ def test_maybe_compress_sample():
     fmt, compressed = maybe_compress(payload, 'lz4')
     assert fmt == None
     assert compressed == payload
+
+
+def test_large_messages():
+    psutil = pytest.importorskip('psutil')
+    pytest.importorskip('lz4')
+    if psutil.virtual_memory().total < 8e9:
+        return
+    msg = {'x': b'0' * (2**31 + 10)}
+
+    b = dumps(msg)
+    msg2 = loads(b)
+    assert msg == msg2
Repository: dask/distributed

Issue:
Results not returned when mapping over an iterator with as_completed
When I map a function over an iterator with `as_completed`, nothing is returned.

``` python3
from distributed import Executor, as_completed

def print_and_return(x):
    print(x)
    return x

e = Executor()
things = iter(range(10))
futures = as_completed(e.map(print_and_return, things))
# Prints out the numbers 0 through 9...
list(futures)
# Returns an empty list!
```

The problem goes away if I map over a list instead:

``` python3
things = list(range(10))
futures = as_completed(e.map(print_and_return, things))
# Prints the numbers 0 through 9...
list(futures)
# Returns a list of 10 futures
```

It's also fine without the `as_completed`:

``` python3
things = iter(range(10))
futures = e.map(print_and_return, things)
# Prints out 0 through 9...
list(futures)
# Returns a list of 10 futures
```

This is on e0b21d1 on Python 3.5.1.



Discussion:

Resolving patch:
diff --git a/distributed/executor.py b/distributed/executor.py
--- a/distributed/executor.py
+++ b/distributed/executor.py
@@ -1593,6 +1593,7 @@ def as_completed(fs):
 
     This function does not return futures in the order in which they are input.
     """
+    fs = list(fs)
     if len(set(f.executor for f in fs)) == 1:
         loop = first(fs).executor.loop
     else:


Test patch:
diff --git a/distributed/tests/test_executor.py b/distributed/tests/test_executor.py
--- a/distributed/tests/test_executor.py
+++ b/distributed/tests/test_executor.py
@@ -3158,3 +3158,11 @@ def test_get_stops_work_after_error(loop):
         sleep(0.01)
     loop2.close(all_fds=True)
     t.join()
+
+
+def test_as_completed_list(loop):
+    with cluster() as (s, [a, b]):
+        with Executor(('127.0.0.1', s['port']), loop=loop) as e:
+            seq = e.map(inc, iter(range(5)))
+            seq2 = list(as_completed(seq))
+            assert set(e.gather(seq2)) == {1, 2, 3, 4, 5}
Repository: dask/distributed

Issue:
OverflowError when sending large sparse arrays
I don't yet have a small reproducible example, but I can make this happen every time I try to collect many large sparse arrays. I do have a notebook that will produce it though, and can make that available. The traceback:

```
Traceback (most recent call last):
  File "/home/jcrist/miniconda/envs/dask_learn/lib/python2.7/site-packages/distributed/core.py", line 266, in write
    frames = protocol.dumps(msg)
  File "/home/jcrist/miniconda/envs/dask_learn/lib/python2.7/site-packages/distributed/protocol.py", line 81, in dumps
    frames = dumps_msgpack(small)
  File "/home/jcrist/miniconda/envs/dask_learn/lib/python2.7/site-packages/distributed/protocol.py", line 155, in dumps_msgpack
    fmt, payload = maybe_compress(payload)
  File "/home/jcrist/miniconda/envs/dask_learn/lib/python2.7/site-packages/distributed/protocol.py", line 137, in maybe_compress
    compressed = compress(payload)
OverflowError: size does not fit in an int
```

A few notes:
- Each array is roughly `675000 x 745`, and ~1% dense. The total bytes for indices + indptr + data is ~40MB each.
- I can get each array individually, so it's not a problem with a chunk being too large
- The error appears only when I'm collecting enough at once (for my size, 39 and and lower works fine).
- At 41 arrays I get the above error, 40 arrays gives me a different (but probably related) error:

```
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-55-7b87709b6c67> in <module>()
----> 1 res = t.compute()

/home/jcrist/dask/dask/base.pyc in compute(self, **kwargs)
     84             Extra keywords to forward to the scheduler ``get`` function.
     85         """
---> 86         return compute(self, **kwargs)[0]
     87 
     88     @classmethod

/home/jcrist/dask/dask/base.pyc in compute(*args, **kwargs)
    177         dsk = merge(var.dask for var in variables)
    178     keys = [var._keys() for var in variables]
--> 179     results = get(dsk, keys, **kwargs)
    180 
    181     results_iter = iter(results)

/home/jcrist/miniconda/envs/dask_learn/lib/python2.7/site-packages/distributed/executor.pyc in get(self, dsk, keys, **kwargs)
   1008 
   1009         if status == 'error':
-> 1010             raise result
   1011         else:
   1012             return result

ValueError: corrupt input at byte 2
```



Discussion:
Did https://github.com/dask/distributed/pull/327 not resolve this?

No, unfortunately.


Resolving patch:
diff --git a/distributed/protocol.py b/distributed/protocol.py
--- a/distributed/protocol.py
+++ b/distributed/protocol.py
@@ -123,6 +123,8 @@ def maybe_compress(payload, compression=default_compression, min_size=1e4,
         return None, payload
     if len(payload) < min_size:
         return None, payload
+    if len(payload) > 2**31:
+        return None, payload
 
     min_size = int(min_size)
     sample_size = int(sample_size)


Test patch:
Repository: dask/distributed

Issue:
Use RSS when tracking memory 
Fixes https://github.com/dask/distributed/issues/1409


Discussion:

Resolving patch:
diff --git a/distributed/worker.py b/distributed/worker.py
--- a/distributed/worker.py
+++ b/distributed/worker.py
@@ -114,6 +114,7 @@ def __init__(self, scheduler_ip, scheduler_port, ip=None, ncores=None,
         self.active = set()
         self.name = name
         self.heartbeat_interval = heartbeat_interval
+        self.heartbeat_active = False
         self._last_disk_io = None
         self._last_net_io = None
         self._ipython_kernel = None
@@ -159,14 +160,19 @@ def __init__(self, scheduler_ip, scheduler_port, ip=None, ncores=None,
 
     @gen.coroutine
     def heartbeat(self):
-        logger.debug("Heartbeat: %s" % self.address)
-        yield self.scheduler.register(address=self.address, name=self.name,
-                                ncores=self.ncores,
-                                now=time(),
-                                info=self.process_health(),
-                                host_info=self.host_health(),
-                                services=self.service_ports,
-                                **self.process_health())
+        if not self.heartbeat_active:
+            self.heartbeat_active = True
+            logger.debug("Heartbeat: %s" % self.address)
+            yield self.scheduler.register(address=self.address, name=self.name,
+                                    ncores=self.ncores,
+                                    now=time(),
+                                    info=self.process_health(),
+                                    host_info=self.host_health(),
+                                    services=self.service_ports,
+                                    **self.process_health())
+            self.heartbeat_active = False
+        else:
+            logger.debug("Heartbeat skipped: channel busy")
 
     @property
     def center(self):


Test patch:
Repository: dask/distributed

Issue:
Can't start worker with --no-nanny
It looks like the dask-worker script makes some assumptions that `t` is a Nanny, when it can be a Worker if `--no-nanny` is given. First symptom: 

> TypeError: **init**() got an unexpected keyword argument 'worker_port'



Discussion:

Resolving patch:
diff --git a/distributed/cli/dask_worker.py b/distributed/cli/dask_worker.py
--- a/distributed/cli/dask_worker.py
+++ b/distributed/cli/dask_worker.py
@@ -76,7 +76,14 @@ def main(scheduler, host, worker_port, http_port, nanny_port, nthreads, nprocs,
     services = {('http', http_port): HTTPWorker}
 
     loop = IOLoop.current()
-    t = Worker if no_nanny else Nanny
+
+    if no_nanny:
+        kwargs = {}
+        t = Worker
+    else:
+        kwargs = {'worker_port': worker_port}
+        t = Nanny
+
     if host is not None:
         ip = socket.gethostbyname(host)
     else:
@@ -84,8 +91,7 @@ def main(scheduler, host, worker_port, http_port, nanny_port, nthreads, nprocs,
         # reach the scheduler
         ip = get_ip(scheduler_ip, scheduler_port)
     nannies = [t(scheduler_ip, scheduler_port, ncores=nthreads, ip=ip,
-                 services=services, name=name, loop=loop,
-                 worker_port=worker_port)
+                 services=services, name=name, loop=loop, **kwargs)
                for i in range(nprocs)]
 
     for nanny in nannies:


Test patch:
diff --git a/distributed/cli/tests/test_dworker.py b/distributed/cli/tests/test_dworker.py
--- a/distributed/cli/tests/test_dworker.py
+++ b/distributed/cli/tests/test_dworker.py
@@ -29,3 +29,10 @@ def test_nanny_worker_ports(loop):
                         assert time() - start < 5
                         sleep(0.1)
                 assert d['workers']['127.0.0.1:8788']['services']['nanny'] == 8789
+
+
+def test_no_nanny(loop):
+    with popen(['dask-scheduler']) as sched:
+        with popen(['dask-worker', '127.0.0.1:8786', '--no-nanny']) as worker:
+            assert any(b'Registered' in worker.stderr.readline()
+                        for i in range(10))
Repository: dask/distributed

Issue:
'FileNotFoundError' is not defined
The scheduler wants to tell me something but in the logs I get 

```
distributed.utils - ERROR - global name 'FileNotFoundError' is not defined
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/distributed/utils.py", line 206, in log_errors
    yield
  File "/usr/local/lib/python2.7/dist-packages/distributed/scheduler.py", line 1572, in worker_stream
    except (StreamClosedError, IOError, OSError, FileNotFoundError):
NameError: global name 'FileNotFoundError' is not defined
```

This is running under Python 2.7.6 and everything was pip-installed.

``` python
In [1]: import distributed
In [2]: distributed.__version__
Out[2]: '1.11.3'
```



Discussion:

Resolving patch:
diff --git a/distributed/scheduler.py b/distributed/scheduler.py
--- a/distributed/scheduler.py
+++ b/distributed/scheduler.py
@@ -1569,7 +1569,7 @@ def worker_stream(self, worker):
 
                     self.ensure_idle_ready()
 
-            except (StreamClosedError, IOError, OSError, FileNotFoundError):
+            except (StreamClosedError, IOError, OSError):
                 logger.info("Worker failed from closed stream: %s", worker)
             finally:
                 if not stream.closed():


Test patch:
Repository: dask/distributed

Issue:
Executor fails when not started
When starting the executor asynchronously, it can happen that tasks are submitted before it has been started, resulting in:

```
Exception raised while submitting task:
'Executor' object has no attribute 'scheduler_stream'
```

It would be good to have a way of querying the executor status, to see if it is currently accepting tasks or not.



Discussion:
That seems reasonable.  We have a `.status` attribute on all other classes.  


Resolving patch:
diff --git a/distributed/executor.py b/distributed/executor.py
--- a/distributed/executor.py
+++ b/distributed/executor.py
@@ -302,6 +302,9 @@ def start(self, **kwargs):
         self.status = 'running'
 
     def _send_to_scheduler(self, msg):
+        if self.status is not 'running':
+            raise Exception("Executor not running.  Status: %s" % self.status)
+
         self.loop.add_callback(self.scheduler_stream.send, msg)
 
     @gen.coroutine
@@ -336,6 +339,7 @@ def _start(self, timeout=3, **kwargs):
 
         _global_executor[0] = self
         yield start_event.wait()
+        self.status = 'running'
         logger.debug("Started scheduling coroutines. Synchronized")
 
     def __enter__(self):
@@ -435,8 +439,8 @@ def _shutdown(self, fast=False):
         """ Send shutdown signal and wait until scheduler completes """
         if self.status == 'closed':
             raise Return()
-        self.status = 'closed'
         self._send_to_scheduler({'op': 'close-stream'})
+        self.status = 'closed'
         if _global_executor[0] is self:
             _global_executor[0] = None
         if not fast:


Test patch:
diff --git a/distributed/tests/test_executor.py b/distributed/tests/test_executor.py
--- a/distributed/tests/test_executor.py
+++ b/distributed/tests/test_executor.py
@@ -3265,3 +3265,24 @@ def create_and_destroy_worker(delay):
             print("Killed nanny")
 
     yield [create_and_destroy_worker(0.1 * i) for i in range(10)]
+
+
+@gen_test()
+def test_status():
+    s = Scheduler()
+    s.start(0)
+
+    e = Executor((s.ip, s.port), start=False)
+    assert e.status != 'running'
+
+    with pytest.raises(Exception):
+        x = e.submit(inc, 1)
+
+    yield e._start()
+    assert e.status == 'running'
+    x = e.submit(inc, 1)
+
+    yield e._shutdown()
+    assert e.status == 'closed'
+
+    yield s.close()
Repository: dask/distributed

Issue:
futures returned from `scatter` should report object's type
I expect the following code

```
a, = client.scatter([5])
print('type of a is', a)
```

to include the 'type' attribute in the future output. What I get is

```
<Future: status: finished, key: 385d6ca61668cb602fe73705b413b0bc>
```



Discussion:

Resolving patch:
diff --git a/distributed/client.py b/distributed/client.py
--- a/distributed/client.py
+++ b/distributed/client.py
@@ -862,14 +862,18 @@ def _scatter(self, data, workers=None, broadcast=False):
 
         if isinstance(data, dict):
             data2 = valmap(dumps, data)
+            types = valmap(type, data)
         elif isinstance(data, (list, tuple, set, frozenset)):
             data2 = list(map(dumps, data))
+            types = list(map(type, data))
         elif isinstance(data, (Iterable, Iterator)):
             data2 = list(map(dumps, data))
+            types = list(map(type, data))
         else:
             raise TypeError("Don't know how to scatter %s" % type(data))
         keys = yield self.scheduler.scatter(data=data2, workers=workers,
-                                            client=self.id, broadcast=broadcast)
+                                            client=self.id,
+                                            broadcast=broadcast)
         if isinstance(data, dict):
             out = {k: Future(k, self) for k in keys}
         elif isinstance(data, (tuple, list, set, frozenset)):
@@ -884,6 +888,13 @@ def _scatter(self, data, workers=None, broadcast=False):
             self.futures[key]['status'] = 'finished'
             self.futures[key]['event'].set()
 
+        if isinstance(types, list):
+            for key, typ in zip(keys, types):
+                self.futures[key]['type'] = typ
+        elif isinstance(types, dict):
+            for key in keys:
+                self.futures[key]['type'] = types[key]
+
         raise gen.Return(out)
 
     def _threaded_scatter(self, q_or_i, qout, **kwargs):


Test patch:
diff --git a/distributed/tests/test_client.py b/distributed/tests/test_client.py
--- a/distributed/tests/test_client.py
+++ b/distributed/tests/test_client.py
@@ -3551,3 +3551,12 @@ def test_get_restrictions():
     r1, loose = get_restrictions(L2, {(total,): '127.0.0.1'}, True)
     assert r1 == {total.key: ['127.0.0.1']}
     assert loose == [total.key]
+
+
+@gen_cluster(client=True)
+def test_scatter_type(c, s, a, b):
+    [future] = yield c._scatter([1])
+    assert future.type == int
+
+    d = yield c._scatter({'x': 1.0})
+    assert d['x'].type == float
Repository: dask/distributed

Issue:
consider truncating huge strings in logging error when deserialization fails
I was getting enormous errors due to this line, which made it difficult to debug. May want to truncate the string logged here if it is huge:

https://github.com/dask/distributed/blob/master/distributed/core.py#L76



Discussion:

Resolving patch:
diff --git a/distributed/core.py b/distributed/core.py
--- a/distributed/core.py
+++ b/distributed/core.py
@@ -73,7 +73,7 @@ def loads(x):
     try:
         return pickle.loads(x)
     except Exception:
-        logger.info("Failed to deserialize %s", x, exc_info=True)
+        logger.info("Failed to deserialize %s", x[:10000], exc_info=True)
         raise
 
 


Test patch:
Repository: dask/distributed

Issue:
Disallow simultaneous scale up and down in Adaptive. Fix for #1604



Discussion:

Resolving patch:
diff --git a/distributed/client.py b/distributed/client.py
--- a/distributed/client.py
+++ b/distributed/client.py
@@ -151,6 +151,15 @@ def exception(self):
         """
         return sync(self.client.loop, self._exception)
 
+    def add_done_callback(self, fn):
+        """ Call callback on future when callback has finished
+
+        The callback ``fn`` should take the future as its only argument.  This
+        will be called regardless of if the future completes successfully,
+        errs, or is cancelled
+        """
+        self.client.loop.add_callback(done_callback, self, fn)
+
     def cancel(self):
         """ Returns True if the future has been cancelled """
         return self.client.cancel([self])
@@ -225,6 +234,14 @@ def __str__(self):
     __repr__ = __str__
 
 
+@gen.coroutine
+def done_callback(future, callback):
+    """ Wait on future, then call callback """
+    while future.status == 'pending':
+        yield future.event.wait()
+    callback(future)
+
+
 @partial(normalize_token.register, Future)
 def normalize_future(f):
     return [f.key, type(f)]


Test patch:
diff --git a/distributed/tests/test_client.py b/distributed/tests/test_client.py
--- a/distributed/tests/test_client.py
+++ b/distributed/tests/test_client.py
@@ -3635,3 +3635,19 @@ def f(x, y=0):
     yield _wait(futures)
 
     assert len(b.data) > 2 * len(a.data)
+
+
+@gen_cluster(client=True)
+def test_add_done_callback(c, s, a, b):
+    x = c.submit(inc, 1)
+
+    L = []
+    def f(future):
+        L.append(future.key)
+        L.append(future.status)
+
+    x.add_done_callback(f)
+
+    yield _wait(x)
+
+    assert L == [x.key, x.status]