Repository: pandas-dev/pandas
Pull request: BUG: hash_pandas_object fails on array containing tuple #28969
- [x] closes #28969
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

ATM this implements a smoke-test that the cases don't raise, but I don't have a canonical answer to test them against.  Could hard-code the result I get now.  Thoughts?

Problem statement:
hash_pandas_object fails on tuple
#### Code Sample

```python
import pandas as pd

pd.util.hash_pandas_object(pd.DataFrame({'data': [tuple('1'), tuple('2')]})) # fails
pd.util.hash_pandas_object(pd.DataFrame({'data': [tuple([1]), tuple([2])]})) # fails
```

<details><summary>Traceback</summary>
<p>

```sh
TypeError                                 Traceback (most recent call last)
~/miniconda3/envs/povmap/lib/python3.7/site-packages/pandas/core/util/hashing.py in hash_array(vals, encoding, hash_key, categorize)
    307         try:
--> 308             vals = hashing.hash_object_array(vals, hash_key, encoding)
    309         except TypeError:

pandas/_libs/hashing.pyx in pandas._libs.hashing.hash_object_array()

TypeError: ('1',) of type  is not a valid type for hashing, must be string or null

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
 in 
      1 import pandas as pd
      2 
----> 3 pd.util.hash_pandas_object(pd.DataFrame({'data': [tuple('1'), tuple('2')]}))

~/miniconda3/envs/povmap/lib/python3.7/site-packages/pandas/core/util/hashing.py in hash_pandas_object(obj, index, encoding, hash_key, categorize)
    129             num_items += 1
    130             hashes = itertools.chain(hashes, index_hash_generator)
--> 131         h = _combine_hash_arrays(hashes, num_items)
    132 
    133         h = Series(h, index=obj.index, dtype="uint64", copy=False)

~/miniconda3/envs/povmap/lib/python3.7/site-packages/pandas/core/util/hashing.py in _combine_hash_arrays(arrays, num_items)
     37     """
     38     try:
---> 39         first = next(arrays)
     40     except StopIteration:
     41         return np.array([], dtype=np.uint64)

~/miniconda3/envs/povmap/lib/python3.7/site-packages/pandas/core/util/hashing.py in (.0)
    114 
    115     elif isinstance(obj, ABCDataFrame):
--> 116         hashes = (hash_array(series.values) for _, series in obj.items())
    117         num_items = len(obj.columns)
    118         if index:

~/miniconda3/envs/povmap/lib/python3.7/site-packages/pandas/core/util/hashing.py in hash_array(vals, encoding, hash_key, categorize)
    303             codes, categories = factorize(vals, sort=False)
    304             cat = Categorical(codes, Index(categories), ordered=False, fastpath=True)
--> 305             return _hash_categorical(cat, encoding, hash_key)
    306 
    307         try:

~/miniconda3/envs/povmap/lib/python3.7/site-packages/pandas/core/util/hashing.py in _hash_categorical(c, encoding, hash_key)
    221     # Convert ExtensionArrays to ndarrays
    222     values = np.asarray(c.categories.values)
--> 223     hashed = hash_array(values, encoding, hash_key, categorize=False)
    224 
    225     # we have uint64, as we don't directly support missing values

~/miniconda3/envs/povmap/lib/python3.7/site-packages/pandas/core/util/hashing.py in hash_array(vals, encoding, hash_key, categorize)
    310             # we have mixed types
    311             vals = hashing.hash_object_array(
--> 312                 vals.astype(str).astype(object), hash_key, encoding
    313             )
    314 

ValueError: setting an array element with a sequence
```
</p>
</details>

#### Problem description
Tuples are immutable and hash should work correct?

#### Expected Output
Hashed dataframe elements.

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit           : None
python           : 3.7.3.final.0
python-bits      : 64
OS               : Linux
OS-release       : 5.0.0-1018-azure
machine          : x86_64
processor        : x86_64
byteorder        : little
LC_ALL           : None
LANG             : C.UTF-8
LOCALE           : en_US.UTF-8

pandas           : 0.25.1
numpy            : 1.17.2
pytz             : 2019.3
dateutil         : 2.8.0
pip              : 19.2.3
setuptools       : 41.4.0
Cython           : None
pytest           : 5.2.1
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : None
html5lib         : None
pymysql          : None
psycopg2         : None
jinja2           : 2.10.3
IPython          : 7.8.0
pandas_datareader: None
bs4              : None
bottleneck       : None
fastparquet      : None
gcsfs            : None
lxml.etree       : None
matplotlib       : 3.1.1
numexpr          : None
odfpy            : None
openpyxl         : 3.0.0
pandas_gbq       : None
pyarrow          : 0.15.0
pytables         : None
s3fs             : None
scipy            : 1.3.1
sqlalchemy       : 1.3.10
tables           : None
xarray           : None
xlrd             : 1.2.0
xlwt             : None
xlsxwriter       : None

</details>

Hints from issue discussion:
Can you include the trace back in the bug report?

In particular we need to isolate whether this is a bug in pandas or numpy
Traceback added.
This looks similar to https://github.com/numpy/numpy/issues/9441.
Ok, how can this be solved? Can I help?
I have no idea about the underlying NumPy issue. You could try posting over there but it seems a bit thorny.

On the pandas side of things, we could perhaps avoid the `.astype(str)` call. May do it in a list comprehension instead?
@TomAugspurger would like to work on this if nobody has taken this issue.
Ok. It’s not clear to me what a correct fix is though. 

> On Oct 21, 2019, at 01:32, Rajhans Jadhao <notifications@github.com> wrote:
> 
> ﻿
> @TomAugspurger would like to work on this if nobody has taken this issue.
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or unsubscribe.

Commit message: BUG: hash_pandas_object fails on array containing tuple #28969

Patch to review:
diff --git a/pandas/core/util/hashing.py b/pandas/core/util/hashing.py
index fa3582755a202..6aba4a7702085 100644
--- a/pandas/core/util/hashing.py
+++ b/pandas/core/util/hashing.py
@@ -85,11 +85,12 @@ def hash_pandas_object(
     if isinstance(obj, ABCMultiIndex):
         return Series(hash_tuples(obj, encoding, hash_key), dtype="uint64", copy=False)
 
-    if isinstance(obj, ABCIndexClass):
+    elif isinstance(obj, ABCIndexClass):
         h = hash_array(obj.values, encoding, hash_key, categorize).astype(
             "uint64", copy=False
         )
         h = Series(h, index=obj, dtype="uint64", copy=False)
+
     elif isinstance(obj, ABCSeries):
         h = hash_array(obj.values, encoding, hash_key, categorize).astype(
             "uint64", copy=False
@@ -306,8 +307,15 @@ def hash_array(
             vals = hashing.hash_object_array(vals, hash_key, encoding)
         except TypeError:
             # we have mixed types
+            try:
+                str_vals = vals.astype(str)
+            except ValueError:
+                # GH#28969 if we contain tuples, astype fails here,
+                #  apparently related to github.com/numpy/numpy/issues/9441
+                str_vals = np.array([str(x) for x in vals], dtype=object)
+
             vals = hashing.hash_object_array(
-                vals.astype(str).astype(object), hash_key, encoding
+                str_vals.astype(object), hash_key, encoding
             )
 
     # Then, redistribute these 64-bit ints within the space of 64-bit ints
diff --git a/pandas/tests/util/test_hashing.py b/pandas/tests/util/test_hashing.py
index ebbdbd6c29842..58c39d150deb9 100644
--- a/pandas/tests/util/test_hashing.py
+++ b/pandas/tests/util/test_hashing.py
@@ -353,3 +353,16 @@ def test_hash_collisions():
 
     result = hash_array(np.asarray(hashes, dtype=object), "utf8")
     tm.assert_numpy_array_equal(result, np.concatenate([expected1, expected2], axis=0))
+
+
+def test_hash_with_tuple():
+    # GH#28969 array containing a tuple raises on call to arr.astype(str)
+    #  apparently a numpy bug github.com/numpy/numpy/issues/9441
+
+    df = pd.DataFrame({"data": [tuple("1"), tuple("2")]})
+
+    hash_pandas_object(df)
+
+    df2 = pd.DataFrame({"data": [tuple([1]), tuple([2])]})
+
+    hash_pandas_object(df2)


Code review comments:
diff_hunk: @@ -306,8 +307,15 @@ def hash_array(
             vals = hashing.hash_object_array(vals, hash_key, encoding)
         except TypeError:
             # we have mixed types
+            try:
+                str_vals = vals.astype(str)
+            except ValueError:
+                # GH#28969 if we contain tuples, astype fails here,
+                #  apparently related to github.com/numpy/numpy/issues/9441
+                str_vals = np.array([str(x) for x in vals], dtype=object)
path: pandas/core/util/hashing.py
text: @user1:
Hmm not sure about this - wouldn't this really make everything hashable at this point, including non-hashable items?

```python
>>> vals = np.array([("a",), ["c", "d"]])
>>> str_vals = np.array([str(x) for x in vals], dtype=object)
array(["('a',)", "['c', 'd']"], dtype=object)
```

@author:
good point

diff_hunk: @@ -306,8 +307,15 @@ def hash_array(
             vals = hashing.hash_object_array(vals, hash_key, encoding)
         except TypeError:
             # we have mixed types
+            try:
+                str_vals = vals.astype(str)
path: pandas/core/util/hashing.py
text: @user1:
i would handle this in hash_object_array as a case