Repository: dask/distributed

Issue:
Array > 2GB hitting `msgpack` limit 
I know that there's supposed to be alternate protocols used for larger arrays but I'm not sure what needs to done to use them (or if they don't play nice with `map_blocks`?

MRE:

``` console
import numpy
import dask.array as da
from distributed import Client

client = Client('127.0.0.1:8786')
def increment_by_one(my_array):
    return my_array + 1

data = numpy.random.random(300000000)
chunks = (10000,)
data = da.from_array(data, chunks=chunks)
output = da.map_blocks(increment_by_one, data)
output.compute()
```

Traceback:

``` console
distributed.utils - ERROR - 2400000161 exceeds max_bin_len(2147483647)
Traceback (most recent call last):
  File "/home/gil/anaconda/envs/dasknumbagpu/lib/python3.5/site-packages/distributed/utils.py", line 207, in log_errors
    yield
  File "/home/gil/anaconda/envs/dasknumbagpu/lib/python3.5/site-packages/distributed/client.py", line 460, in _handle_report
    six.reraise(*clean_exception(**msg))
  File "/home/gil/anaconda/envs/dasknumbagpu/lib/python3.5/site-packages/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/home/gil/anaconda/envs/dasknumbagpu/lib/python3.5/site-packages/tornado/concurrent.py", line 237, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 3, in raise_exc_info
  File "/home/gil/anaconda/envs/dasknumbagpu/lib/python3.5/site-packages/tornado/gen.py", line 1024, in run
    yielded = self.gen.send(value)
  File "/home/gil/anaconda/envs/dasknumbagpu/lib/python3.5/site-packages/distributed/core.py", line 258, in read
    msg = protocol.loads(frames)
  File "/home/gil/anaconda/envs/dasknumbagpu/lib/python3.5/site-packages/distributed/protocol.py", line 152, in loads
    msg = loads_msgpack(small_header, small_payload)
  File "/home/gil/anaconda/envs/dasknumbagpu/lib/python3.5/site-packages/distributed/protocol.py", line 256, in loads_msgpack
    return msgpack.loads(payload, encoding='utf8')
  File "pandas/msgpack/_unpacker.pyx", line 138, in pandas.msgpack._unpacker.unpackb (pandas/msgpack/_unpacker.cpp:2059)
ValueError: 2400000161 exceeds max_bin_len(2147483647)
```



Discussion:
I can reproduce the issue with the following slightly simpler example

``` python
In [1]: import numpy
   ...: import dask.array as da
   ...: from distributed import Client
   ...: 
   ...: client = Client()
   ...: 
   ...: data = numpy.random.random(300000000)
   ...: chunks = (10000,)
   ...: data = da.from_array(data, chunks=chunks)
   ...: data = client.persist(data)
```

So there are a couple things going on here:

### Protocol error

Apparently some large bytestrings are able to slip through the protocol (which usually cleans them out and sends them separately) and into msgpack, which as you've noted, can't handle them.  This needs to be fixed internally

### User behaviors

Generally we should avoid putting large amounts of data in the dask graph.  Now of course, it's note entirely clear when you're doing this.  This happens whenever you use a dask.array function that consumes a large numpy array and gives you back a dask.array.  If you look at the `.dask` property afterwards you'll notice that there are numpy arrays within it.  This `.dask` attribute gets sent up to the scheduler, which we generally want to keep fairly light, so our goal is to make it so that data stays out of the `.dask` graph, either by using loading functions (like da.random.random or da.from_delayed) or futures, both of which are shown below.  This is important for performance even after we fix the protocol error above.

It's best to load data direction on the workers if possible.  This can be done with dask.array functions like `da.random.random`, or with client.submit/map or with dask.delayed.  This doc page might be interesting: http://distributed.readthedocs.io/en/latest/memory.html

``` python
x = da.random.random(size=..., chunks=...)
```

or 

``` python
data = e.submit(np.random.random, size)   # produces a single future
```

or 

``` python
data = np.random.random(...)
future = client.scatter(data)  # produces a single future
```

From this future you can construct a dask.array of a single chunk and then rechunk it.  Here is a comprehensive example using scatter:

``` python
In [1]: from distributed import Client

In [2]: client = Client()

In [3]: import numpy as np

In [4]: data = np.random.random(10000000)

In [5]: data.nbytes / 1e6
Out[5]: 80.0

In [6]: %time [future] = client.scatter([data])
CPU times: user 1.08 s, sys: 132 ms, total: 1.21 s
Wall time: 1.25 s

In [7]: import dask.array as da

In [8]: from dask import delayed

In [22]: x = da.from_delayed(delayed(future), shape=data.shape, dtype=data.dtype)

In [23]: x
Out[23]: dask.array<from-va..., shape=(10000000,), dtype=float64, chunksize=(10000000,)>

In [24]: x = x.rechunk((30000,))

In [25]: x
Out[25]: dask.array<rechunk..., shape=(10000000,), dtype=float64, chunksize=(30000,)>

In [26]: x = client.persist(x)

In [28]: {w: len(keys) for w, keys in client.has_what().items()}  # but wait!  all data on the same worker
Out[28]: 
{'25.231.111.255:35376': 0,
 '25.231.111.255:38110': 335,
 '25.231.111.255:42691': 0,
 '25.231.111.255:45072': 0}

In [29]: client.rebalance(x)

In [30]: {w: len(keys) for w, keys in client.has_what().items()}
Out[30]: 
{'25.231.111.255:35376': 82,
 '25.231.111.255:38110': 85,
 '25.231.111.255:42691': 84,
 '25.231.111.255:45072': 84}
```

However this may not be your actual problem.  I've just optimized the fail case you've provided.  This may not be the best approach to what you actually want to do, so it's best to take this all with some salt.

OK, so the protocol is failing here because its a single message of very many small parts.  

None of the chunks are individually large enough to warrant being pulled out into a separate frame and yet when taken all together the message is long enough to break msgpack.  I didn't realize that the 2GB msgpack limit wasn't per-value in the message but rather a limit over the entire message.  

This could become problematic in other cases as well.  I don't currently see a simple fix other than to better inform users on how to avoid these situations.   We would need to become better at picking apart arbitrary messages and then merging them back together downstream.  This isn't trivial to do generally over all possible msgpack messages.

cc @jcrist in case he has ideas


Resolving patch:
diff --git a/distributed/protocol.py b/distributed/protocol.py
--- a/distributed/protocol.py
+++ b/distributed/protocol.py
@@ -110,8 +110,8 @@ def _extract_big_bytes(x, big, path=()):
 def dumps(msg):
     """ Transform Python value to bytestream suitable for communication """
     big = {}
-    # Only dicts can contain big values
-    if isinstance(msg, dict):
+    # Only lists and dicts can contain big values
+    if isinstance(msg, (list, dict)):
         msg, big = extract_big_bytes(msg)
     small_header, small_payload = dumps_msgpack(msg)
     if not big:


Test patch:
diff --git a/distributed/tests/test_protocol.py b/distributed/tests/test_protocol.py
--- a/distributed/tests/test_protocol.py
+++ b/distributed/tests/test_protocol.py
@@ -115,3 +115,13 @@ def f(n):
     big_header = msgpack.loads(b[2], encoding='utf8')
     assert len(big_header['shards']) == 2
     assert len(big_header['keys']) + 2 + 1 == len(b)
+
+    msg = [big_bytes, {'x': big_bytes, 'y': b'small_bytes'}]
+    b = dumps(msg)
+    msg2 = loads(b)
+    assert msg == msg2
+
+    assert len(b) >= 2
+    big_header = msgpack.loads(b[2], encoding='utf8')
+    assert len(big_header['shards']) == 2
+    assert len(big_header['keys']) + 2 + 1 == len(b)