Repository: pandas-dev/pandas

Issue:
hash_pandas_object on ExtensionArray-backed Series fails with TypeError
```python
In [1]: import pandas as pd

In [2]: pd.Series
Out[2]: pandas.core.series.Series

In [4]: s = pd.Series(pd.interval_range(0, periods=10))

In [5]: pd.util.hash_pandas_object(s)
```

```pytb
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-5-1b7247db4f16> in <module>
----> 1 pd.util.hash_pandas_object(s)

~/sandbox/pandas/pandas/core/util/hashing.py in hash_pandas_object(obj, index, encoding, hash_key, categorize)
     88     elif isinstance(obj, ABCSeries):
     89         h = hash_array(obj.values, encoding, hash_key,
---> 90                        categorize).astype('uint64', copy=False)
     91         if index:
     92             index_iter = (hash_pandas_object(obj.index,

~/sandbox/pandas/pandas/core/util/hashing.py in hash_array(vals, encoding, hash_key, categorize)
    269     # we'll be working with everything as 64-bit values, so handle this
    270     # 128-bit value early
--> 271     elif np.issubdtype(dtype, np.complex128):
    272         return hash_array(vals.real) + 23 * hash_array(vals.imag)
    273

~/Envs/pandas-dev/lib/python3.7/site-packages/numpy/core/numerictypes.py in issubdtype(arg1, arg2)
    712     """
    713     if not issubclass_(arg1, generic):
--> 714         arg1 = dtype(arg1).type
    715     if not issubclass_(arg2, generic):
    716         arg2_orig = arg2

TypeError: data type not understood

In [6]: s
Out[6]:
0     (0, 1]
1     (1, 2]
2     (2, 3]
3     (3, 4]
4     (4, 5]
5     (5, 6]
6     (6, 7]
7     (7, 8]
8     (8, 9]
9    (9, 10]
dtype: interval
```

Options

1. convert to object before hashing
2. add some kind of `_hash_values` to the interface. But, how do we prevent hash collisions between similar, but different EAs? For example, the fastest hash for a `PeriodArray` would be to just hash the ordinals. But we wouldn't want the following two to hash identically (using my PeriodArray branch)

```
In [35]: pd.core.arrays.PeriodArray._from_ordinals([10, 20], freq='H')
Out[35]:
<pandas PeriodArray>
['1970-01-01 10:00', '1970-01-01 20:00']
Length: 2, dtype: period[H]

In [36]: pd.core.arrays.PeriodArray._from_ordinals([10, 20], freq='D')
Out[36]:
<pandas PeriodArray>
['1970-01-11', '1970-01-21']
Length: 2, dtype: period[D]
```

So we need to mix the dtype information in too.


Discussion:
Slightly related, this is an issue on with datetime / datetimetz

```python
In [86]: ns = np.array([946706400000000000, 946792800000000000, 946879200000000000,
    ...:        946965600000000000])
    ...:

In [87]: a = pd.Series(pd.DatetimeIndex(ns))

In [88]: b = pd.Series(pd.DatetimeIndex(ns, tz='UTC'))

In [89]: pd.util.testing.assert_series_equal(pd.util.hash_pandas_object(a), pd.util.hash_pandas_object(b))
```
Hmm, maybe we just don't care about different dtypes having the same hashed values (which is perfectly fair)

```python
In [92]: a = pd.Series(pd.Categorical([0, 0, 1, 2]))

In [93]: b = pd.Series([0, 0, 1, 2])

In [94]: pd.util.hash_pandas_object(a)
Out[94]:
0    3713087409444908179
1    7478705303072568462
2    3975671353655200382
3    3563156779521628949
dtype: uint64

In [95]: pd.util.hash_pandas_object(b)
Out[95]:
0    3713087409444908179
1    7478705303072568462
2    3975671353655200382
3    3563156779521628949
dtype: uint64
```
iirc ther is an issue about the datetime hashing from a while ago 
Looks like https://github.com/pandas-dev/pandas/issues/16372

So we have two issues

1. An extension point for EAs to determine how they're hashed.
2. How to avoid "collisions" between different dtypes (#16372)

Let's focus on the first issue here. I think EAs need some kind of way to say what values are hashed. Performance seems too critical to just `.astype(object)` here. So two options

1. A new `_values_for_hashing` method
2. Overload `_values_for_factorize` and use that

Right now I'm leaning toward 2.
> Performance seems too critical to just .astype(object) here. So two options

Also, `astype(object)` does not necessarily give you hashable values (eg won't be the case for geometries)

What would be the reason not to use `_values_for_factorize`? They already need to be hashable, and should be unique to the original data they represent (since they are round-trippable)
re-using _values_for_factorize should be fine.

Resolving patch:
diff --git a/pandas/core/arrays/base.py b/pandas/core/arrays/base.py
--- a/pandas/core/arrays/base.py
+++ b/pandas/core/arrays/base.py
@@ -466,6 +466,11 @@ def _values_for_factorize(self):
             as NA in the factorization routines, so it will be coded as
             `na_sentinal` and not included in `uniques`. By default,
             ``np.nan`` is used.
+
+        Notes
+        -----
+        The values returned by this method are also used in
+        :func:`pandas.util.hash_pandas_object`.
         """
         return self.astype(object), np.nan
 
diff --git a/pandas/core/util/hashing.py b/pandas/core/util/hashing.py
--- a/pandas/core/util/hashing.py
+++ b/pandas/core/util/hashing.py
@@ -11,7 +11,7 @@
     ABCSeries,
     ABCDataFrame)
 from pandas.core.dtypes.common import (
-    is_categorical_dtype, is_list_like)
+    is_categorical_dtype, is_list_like, is_extension_array_dtype)
 from pandas.core.dtypes.missing import isna
 from pandas.core.dtypes.cast import infer_dtype_from_scalar
 
@@ -265,10 +265,13 @@ def hash_array(vals, encoding='utf8', hash_key=None, categorize=True):
     # numpy if categorical is a subdtype of complex, as it will choke).
     if is_categorical_dtype(dtype):
         return _hash_categorical(vals, encoding, hash_key)
+    elif is_extension_array_dtype(dtype):
+        vals, _ = vals._values_for_factorize()
+        dtype = vals.dtype
 
     # we'll be working with everything as 64-bit values, so handle this
     # 128-bit value early
-    elif np.issubdtype(dtype, np.complex128):
+    if np.issubdtype(dtype, np.complex128):
         return hash_array(vals.real) + 23 * hash_array(vals.imag)
 
     # First, turn whatever array this is into unsigned 64-bit ints, if we can


Test patch:
diff --git a/pandas/tests/extension/base/base.py b/pandas/tests/extension/base/base.py
--- a/pandas/tests/extension/base/base.py
+++ b/pandas/tests/extension/base/base.py
@@ -2,6 +2,7 @@
 
 
 class BaseExtensionTests(object):
+    assert_equal = staticmethod(tm.assert_equal)
     assert_series_equal = staticmethod(tm.assert_series_equal)
     assert_frame_equal = staticmethod(tm.assert_frame_equal)
     assert_extension_array_equal = staticmethod(
diff --git a/pandas/tests/extension/base/methods.py b/pandas/tests/extension/base/methods.py
--- a/pandas/tests/extension/base/methods.py
+++ b/pandas/tests/extension/base/methods.py
@@ -164,3 +164,13 @@ def test_container_shift(self, data, frame, periods, indices):
             compare = self.assert_series_equal
 
         compare(result, expected)
+
+    @pytest.mark.parametrize("as_frame", [True, False])
+    def test_hash_pandas_object_works(self, data, as_frame):
+        # https://github.com/pandas-dev/pandas/issues/23066
+        data = pd.Series(data)
+        if as_frame:
+            data = data.to_frame()
+        a = pd.util.hash_pandas_object(data)
+        b = pd.util.hash_pandas_object(data)
+        self.assert_equal(a, b)
diff --git a/pandas/tests/extension/json/test_json.py b/pandas/tests/extension/json/test_json.py
--- a/pandas/tests/extension/json/test_json.py
+++ b/pandas/tests/extension/json/test_json.py
@@ -199,6 +199,10 @@ def test_combine_le(self, data_repeated):
     def test_combine_add(self, data_repeated):
         pass
 
+    @unhashable
+    def test_hash_pandas_object_works(self, data, kind):
+        super().test_hash_pandas_object_works(data, kind)
+
 
 class TestCasting(BaseJSON, base.BaseCastingTests):
     @pytest.mark.skip(reason="failing on np.array(self, dtype=str)")