Repository: dask/dask

Issue:
Bug (?) concatenating arrays of strings
Hi,

I've run across what I think is a small bug with concatenating dask arrays of strings in which the dtypes of the arrays to be concatenated are different:

```
In [106]: a = np.array(['CA-0', 'CA-1'])

In [107]: b = np.array(['TX-0', 'TX-10', 'TX-101', 'TX-102'])

In [108]: a = da.from_array(a, chunks=2)

In [109]: b = da.from_array(b, chunks=4)

In [110]: da.concatenate([a, b]).compute()
Out[110]: 
array(['CA-0', 'CA-1', 'TX-0', 'TX-1', 'TX-1', 'TX-1'],
      dtype='|S4')

In [111]: da.concatenate([b, a]).compute()
Out[111]: 
array(['TX-0', 'TX-10', 'TX-101', 'TX-102', 'CA-0', 'CA-1'],
      dtype='|S6')
```

If the array with the "smaller" dtype (in this case, S4) is the first array in the sequence to be concatenated, then this "smaller" dtype is used for the end result, truncating the entries in the array with the "larger" dtype (in this case, S6). If the order of the arrays is swapped so that the array with the "larger" dtype comes first, then the concatenation works properly.

It looks to me like the error occurs in the [dask.array.core.concatenate3](https://github.com/dask/dask/blob/master/dask/array/core.py#L2952) function where the dtype of the result is inferred from the first array in the sequence, rather than using the dtype computed in the [concatenate](https://github.com/dask/dask/blob/master/dask/array/core.py#L1748) function itself.

Todd



Discussion:
Hi @ToddSmall!

This is probably a good example of how it can be convenient to cast arrays of strings to `dtype=object` (like pandas) before doing operations.

Yeah, I think it will be challenging to lazily determine dtypes when the dtypes depend on runtime values.  Going with object dtype or explicitly casting to a large enough dtype manually seems to me to be the way to go.

Well, dask is looking at argument dtypes to infer the dtype of the result,
so there's no reason why it couldn't follow the promotion rules of numpy
here when combining arrays.

On Mon, Mar 14, 2016 at 3:57 PM, Matthew Rocklin notifications@github.com
wrote:

> Yeah, I think it will be challenging to lazily determine dtypes when the
> dtypes depend on runtime values. Going with object dtype or explicitly
> casting to a large enough dtype manually seems to me to be the way to go.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/dask/dask/issues/1050#issuecomment-196556104.

Oh indeed.  Actually, if I looked longer at @ToddSmall 's original comment I probably would have seen his astute description of where the problem lies.  My bad.

Thanks @shoyer and @mrocklin. I'll use one or the other of those suggestions.

We should definitely fix this though.  I suggest that in `da.concatenate`, if the output dtype is not the same as the input dtypes, then we first call `astype` on all inputs with the new output dtype.

@shoyer can you anticipate any occasion in which this might fail?


Resolving patch:
diff --git a/dask/array/core.py b/dask/array/core.py
--- a/dask/array/core.py
+++ b/dask/array/core.py
@@ -1045,6 +1045,8 @@ def topk(self, k):
 
     def astype(self, dtype, **kwargs):
         """ Copy of the array, cast to a specified type """
+        if dtype == self._dtype:
+            return self
         name = tokenize('astype', self, dtype, kwargs)
         return elemwise(lambda x: x.astype(dtype, **kwargs), self,
                         dtype=dtype, name=name)
@@ -1798,6 +1800,13 @@ def concatenate(seq, axis=0):
               + seq[0].chunks[axis + 1:])
 
     cum_dims = [0] + list(accumulate(add, [len(a.chunks[axis]) for a in seq]))
+
+    if all(a._dtype is not None for a in seq):
+        dt = reduce(np.promote_types, [a._dtype for a in seq])
+        seq = [x.astype(dt) for x in seq]
+    else:
+        dt = None
+
     names = [a.name for a in seq]
 
     name = 'concatenate-' + tokenize(names, axis)
@@ -1812,10 +1821,7 @@ def concatenate(seq, axis=0):
     dsk = dict(zip(keys, values))
     dsk2 = merge(dsk, *[a.dask for a in seq])
 
-    if all(a._dtype is not None for a in seq):
-        dt = reduce(np.promote_types, [a._dtype for a in seq])
-    else:
-        dt = None
+
 
     return Array(dsk2, name, chunks, dtype=dt)
 
diff --git a/dask/dataframe/io.py b/dask/dataframe/io.py
--- a/dask/dataframe/io.py
+++ b/dask/dataframe/io.py
@@ -565,11 +565,11 @@ def from_dask_array(x, columns=None):
     >>> x = da.ones((4, 2), chunks=(2, 2))
     >>> df = dd.io.from_dask_array(x, columns=['a', 'b'])
     >>> df.compute()
-       a  b
-    0  1  1
-    1  1  1
-    2  1  1
-    3  1  1
+         a    b
+    0  1.0  1.0
+    1  1.0  1.0
+    2  1.0  1.0
+    3  1.0  1.0
     """
 
     dummy = _dummy_from_array(x, columns)


Test patch:
diff --git a/dask/array/tests/test_array_core.py b/dask/array/tests/test_array_core.py
--- a/dask/array/tests/test_array_core.py
+++ b/dask/array/tests/test_array_core.py
@@ -285,6 +285,17 @@ def test_concatenate():
     assert raises(ValueError, lambda: concatenate([a, b, c], axis=2))
 
 
+def test_concatenate_fixlen_strings():
+    x = np.array(['a', 'b', 'c'])
+    y = np.array(['aa', 'bb', 'cc'])
+
+    a = da.from_array(x, chunks=(2,))
+    b = da.from_array(y, chunks=(2,))
+
+    assert_eq(np.concatenate([x, y]),
+              da.concatenate([a, b]))
+
+
 def test_vstack():
     x = np.arange(5)
     y = np.ones(5)
@@ -967,6 +978,8 @@ def test_astype():
     assert_eq(d.astype('i8'), x.astype('i8'))
     assert same_keys(d.astype('i8'), d.astype('i8'))
 
+    assert d.astype(d.dtype) is d
+
 
 def test_arithmetic():
     x = np.arange(5).astype('f4') + 2