Repository: dask/dask

Issue:
Encoding issues when converting a dataframe from pd.read_pickle using dd.from_pandas
Hello,

I have a large pandas data frame (more than 10 thousand columns) that I have saved as a pickle using the pandas API. I want to process this information using dask capabilities but for doing that I  first need to transform it into a dask data frame. 

Dask has a function called dd.from_pandas that should be the solution, but after loading my data using `df = pd.read_pickle(*.pckl)` and then trying to convert it into a dask data-frame  with `df = dd.from_pandas(df, npartitions=4)`, I get the following error:

```
UnicodeEncodeError: 'utf-8' codec can't encode character '\ud83d' in position 191784: surrogates not allowed
```
I have tried to check the documentation if there's a way I can change the encoding in Dask, but have not found a way around. As I have so many columns, I will prefer not to fix this encoding issues column by column. Is there a way to force an Ascii encoding?

Thanks! 

I'm using Python 3 and Dask 0.15.2


Discussion:
Can you provide a full traceback?  Are you able to provide a minimal
reproducible example?

On Mon, Sep 25, 2017 at 8:19 AM, LeonardoReyes <notifications@github.com>
wrote:

> Hello,
>
> I have a large pandas data frame (more than 10 thousand columns) that I
> have saved as a pickle using the pandas API. I want to process this
> information using dask capabilities but for doing that I first need to
> transform it into a dask data frame.
>
> Dask has a function called dd.from_pandas that should be the solution, but
> after loading my data using df = pd.read_pickle(*.pckl) and then trying
> to convert it into a dask data-frame with df = dd.from_pandas(df,
> npartitions=4), I get the following error:
>
> UnicodeEncodeError: 'utf-8' codec can't encode character '\ud83d' in position 191784: surrogates not allowed
>
> I have tried to check the documentation if there's a way I can change the
> encoding in Dask, but have not found a way around. As I have so many
> columns, I will prefer not to fix this encoding issues column by column. Is
> there a way to force an Ascii encoding?
>
> Thanks!
>
> I'm using Python 3 and Dask 0.15.2
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/dask/issues/2713>, or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AASszBXLMBEq0BxbCQ2MTIsRB7rXc18Mks5sl5ppgaJpZM4Pio4y>
> .
>

Yes sure!

Reproducible example:
```
import numpy as np
import pandas as pd
from dask import dataframe as dd 

df=pd.DataFrame(data=np.random.random((3, 1)),
                columns=['\ud83d'])
dd.from_pandas(df, npartitions=4)
```



Traceback:
```
---------------------------------------------------------------------------
UnicodeEncodeError                        Traceback (most recent call last)
<ipython-input-8-2efcd6adb34e> in <module>()
----> 1 df = dd.from_pandas(df, npartitions=4)
      2 
      3 # df_train = ...           # preprocess data
      4 # labels_train = ...
      5 df.info()

/usr/local/lib/python3.5/dist-packages/dask/dataframe/io/io.py in from_pandas(data, npartitions, chunksize, sort, name)
    179         npartitions = int(ceil(nrows / chunksize))
    180 
--> 181     name = name or ('from_pandas-' + tokenize(data, chunksize))
    182 
    183     if not nrows:

/usr/local/lib/python3.5/dist-packages/dask/base.py in tokenize(*args, **kwargs)
    429     if kwargs:
    430         args = args + (kwargs,)
--> 431     return md5(str(tuple(map(normalize_token, args))).encode()).hexdigest()
    432 
    433 

/usr/local/lib/python3.5/dist-packages/dask/utils.py in __call__(self, arg)
    413         """
    414         meth = self.dispatch(type(arg))
--> 415         return meth(arg)
    416 
    417 

/usr/local/lib/python3.5/dist-packages/dask/base.py in normalize_dataframe(df)
    373         data = [block.values for block in df._data.blocks]
    374         data += [df.columns, df.index]
--> 375         return list(map(normalize_token, data))
    376 
    377 

/usr/local/lib/python3.5/dist-packages/dask/utils.py in __call__(self, arg)
    413         """
    414         meth = self.dispatch(type(arg))
--> 415         return meth(arg)
    416 
    417 

/usr/local/lib/python3.5/dist-packages/dask/base.py in normalize_index(ind)
    355     @normalize_token.register(pd.Index)
    356     def normalize_index(ind):
--> 357         return [ind.name, normalize_token(ind.values)]
    358 
    359     @normalize_token.register(pd.Categorical)

/usr/local/lib/python3.5/dist-packages/dask/utils.py in __call__(self, arg)
    413         """
    414         meth = self.dispatch(type(arg))
--> 415         return meth(arg)
    416 
    417 

/usr/local/lib/python3.5/dist-packages/dask/base.py in normalize_array(x)
    394         if x.dtype.hasobject:
    395             try:
--> 396                 data = hash_buffer_hex('-'.join(x.flat).encode('utf-8'))
    397             except TypeError:
    398                 data = hash_buffer_hex(b'-'.join([unicode(item).encode('utf-8') for item in

UnicodeEncodeError: 'utf-8' codec can't encode character '\ud83d' in position 191784: surrogates not allowed
```


I think the problem is because that character is an emoji. Pandas deals with them but maybe not when pushing it to Dask? 

I loaded my pandas df and got a slice of the last ten columns got this 

['💑', '💜💜💜', '📷', '📸', '😀😁😂', '😁🐶', '😂', '😉.', '🙁', '🙂']

did a df.info and got this:

```
<class 'pandas.core.frame.DataFrame'>
Int64Index: 1136 entries, 0 to 1135
Columns: 22055 entries, 'cause to 🙌🏽
dtypes: float64(22053), int64(1), object(1)
memory usage: 191.2+ MB
```
so emojis seem to be handled, but the problem is when I push that df to a dd using dd.from_pandas.
@LeonardoReyes The same `UnicodeEncodeError: 'utf-8' codec can't encode character '\ud83d' in position 191784: surrogates not allowed` error will be raised if you simply try to display the dataframe you created in your example. It seems the error is not specific to dask and the issue should probably be opened in pandas. For your example dataframe this should work: `df.columns=[col.encode('utf-8', 'replace').decode('utf-8') for col in df.columns]` 




From running the example it looks like the exception comes from these lines:

https://github.com/dask/dask/blob/e9731f7810706fd60391d38f65c947b27b6cd548/dask/base.py#L729-L734

Which is called from here 

https://github.com/dask/dask/blob/e9731f7810706fd60391d38f65c947b27b6cd548/dask/base.py#L706-L710

This part of the codebase is about hashing down a pandas object efficiently.  It looks like our mechanism to hash down a numpy array of unicode characters may be broken (this is the first code cell above).  If anyone has suggestions on how that should be done (keeping efficiency in mind) then that would be a welcome PR.
I'm on it ⏳ 

Resolving patch:
diff --git a/dask/base.py b/dask/base.py
--- a/dask/base.py
+++ b/dask/base.py
@@ -729,12 +729,13 @@ def normalize_array(x):
                     x.shape, x.strides, offset)
         if x.dtype.hasobject:
             try:
-                # string fast-path
-                data = hash_buffer_hex('-'.join(x.flat).encode(encoding='utf-8', errors='surrogatepass'))
-            except UnicodeDecodeError:
-                # bytes fast-path
-                data = hash_buffer_hex(b'-'.join(x.flat))
-            except TypeError:
+                try:
+                    # string fast-path
+                    data = hash_buffer_hex('-'.join(x.flat).encode(encoding='utf-8', errors='surrogatepass'))
+                except UnicodeDecodeError:
+                    # bytes fast-path
+                    data = hash_buffer_hex(b'-'.join(x.flat))
+            except (TypeError, UnicodeDecodeError):
                 # object data w/o fast-path, use fast cPickle
                 try:
                     data = hash_buffer_hex(cPickle.dumps(x, cPickle.HIGHEST_PROTOCOL))


Test patch:
diff --git a/dask/tests/test_base.py b/dask/tests/test_base.py
--- a/dask/tests/test_base.py
+++ b/dask/tests/test_base.py
@@ -214,6 +214,12 @@ def test_tokenize_pandas_invalid_unicode():
     tokenize(df)
 
 
+@pytest.mark.skipif('not pd')
+def test_tokenize_pandas_mixed_unicode_bytes():
+    df = pd.DataFrame({u'ö'.encode('utf8'): [1, 2, 3], u'ö': [u'ö', u'ö'.encode('utf8'), None]}, index=[1, 2, 3])
+    tokenize(df)
+
+
 @pytest.mark.skipif('not pd')
 def test_tokenize_pandas_no_pickle():
     class NoPickle(object):