Repository: dask/distributed

Issue:
Cannot serialize datetime64/timedelta64 numpy arrays
```
distributed.core - INFO - Unserializable Message: {'finalize-c29ed21fd1e79d8eb6b3ff89cf2ae354': <Serialize: ['2010-01-01T00:00:00.000000000' '2010-01-02T00:00:00.000000000'
 '2010-01-03T00:00:00.000000000']>}
distributed.core - ERROR - cannot include dtype 'M' in a buffer
Traceback (most recent call last):
  File "/Users/shoyer/dev/distributed/distributed/core.py", line 228, in write
    frames = protocol.dumps(msg)
  File "/Users/shoyer/dev/distributed/distributed/protocol/core.py", line 43, in dumps
    for key, value in data.items()
  File "/Users/shoyer/dev/distributed/distributed/protocol/core.py", line 44, in <dictcomp>
    if type(value) is Serialize}
  File "/Users/shoyer/dev/distributed/distributed/protocol/serialize.py", line 103, in serialize
    header, frames = serializers[name](x)
  File "/Users/shoyer/dev/distributed/distributed/protocol/numpy.py", line 58, in serialize_numpy_ndarray
    frames = [x.data]
ValueError: cannot include dtype 'M' in a buffer
```

The buffer protocol fails on these (sigh). You can view as int64, but then you'll need to keep track of the original type (including time units) separately.

This came up in some of my integration tests with xarray.


Discussion:
Thanks for the bug report.  We've been playing catch up with numpy dtypes ever since the serialization change.  See https://github.com/dask/distributed/pull/630

Should be doable to fix.  Can you think of any other cases that might arise that are not yet tested?

https://github.com/dask/distributed/blob/master/distributed/protocol/tests/test_numpy.py

Looks like you're also missing complex and boolean.

Thanks.  Fixing datetimes now.  Common case seems trivial (view doesn't copy, right?).  Struct case is slightly more of a pain.


Resolving patch:
diff --git a/distributed/protocol/numpy.py b/distributed/protocol/numpy.py
--- a/distributed/protocol/numpy.py
+++ b/distributed/protocol/numpy.py
@@ -47,15 +47,17 @@ def serialize_numpy_ndarray(x):
               'strides': x.strides,
               'shape': x.shape}
 
+    data = x.view('u1').data
+
     if blosc:
-        frames = frame_split_size([x.data])
+        frames = frame_split_size([data])
         if sys.version_info.major == 2:
             frames = [ensure_bytes(frame) for frame in frames]
         frames = [blosc.compress(frame, typesize=size,
                                  cname='lz4', clevel=5) for frame in frames]
         header['compression'] = ['blosc'] * len(frames)
     else:
-        frames = [x.data]
+        frames = [data]
 
     header['lengths'] = [x.nbytes]
 


Test patch:
diff --git a/distributed/protocol/tests/test_numpy.py b/distributed/protocol/tests/test_numpy.py
--- a/distributed/protocol/tests/test_numpy.py
+++ b/distributed/protocol/tests/test_numpy.py
@@ -28,8 +28,19 @@ def test_serialize():
 
 @pytest.mark.parametrize('x',
         [np.ones(5),
+         np.array(5),
          np.asfortranarray(np.random.random((5, 5))),
          np.random.random(5).astype('f4'),
+         np.random.random(5).astype('>i8'),
+         np.random.random(5).astype('<i8'),
+         np.arange(5).astype('M8[us]'),
+         np.arange(5).astype('M8[ms]'),
+         np.arange(5).astype('m8'),
+         np.arange(5).astype('m8[s]'),
+         np.arange(5).astype('c16'),
+         np.arange(5).astype('c8'),
+         np.array([True, False, True]),
+         np.ones(shape=5, dtype=[('a', 'i4'), ('b', 'M8[us]')]),
          np.array(['abc'], dtype='S3'),
          np.array(['abc'], dtype='U3'),
          np.array(['abc'], dtype=object),
@@ -75,14 +86,17 @@ def test_dumps_serialize_numpy_large():
 
 @pytest.mark.parametrize('dt,size', [('f8', 8),
                                      ('i4', 4),
+                                     ('c16', 16),
+                                     ('b', 1),
                                      ('S3', 3),
+                                     ('M8[us]', 8),
+                                     ('M8[s]', 8),
                                      ('U3', 12),
                                      ([('a', 'i4'), ('b', 'f8')], 12),
                                      (('i4', 100), 4),
                                      ([('a', 'i4', 100)], 8),
                                      ([('a', 'i4', 20), ('b', 'f8')], 20*4 + 8),
                                      ([('a', 'i4', 200), ('b', 'f8')], 8)])
-
 def test_itemsize(dt, size):
     assert itemsize(np.dtype(dt)) == size