Repository: great-expectations/great_expectations

Issue:
unhashable type: 'dict' on files with nested map/object
## Problem

If you try to load an object with nested `dict` (JSON object, or map/Struct in Parquet), on Pandas computation engine, will raise this error (the same data is fine with PySpark):

```
Traceback (most recent call last):
  File "/home/shahin/Work/Experiment/ge_app/.direnv/python-3.7.2/bin/great_expectations", line 8, in <module>
    sys.exit(main())
  File "/home/shahin/Work/Experiment/ge_app/.direnv/python-3.7.2/lib/python3.7/site-packages/great_expectations/cli/cli.py", line 66, in main
    cli()
  File "/home/shahin/Work/Experiment/ge_app/.direnv/python-3.7.2/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/shahin/Work/Experiment/ge_app/.direnv/python-3.7.2/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/shahin/Work/Experiment/ge_app/.direnv/python-3.7.2/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/shahin/Work/Experiment/ge_app/.direnv/python-3.7.2/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/shahin/Work/Experiment/ge_app/.direnv/python-3.7.2/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/shahin/Work/Experiment/ge_app/.direnv/python-3.7.2/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/shahin/Work/Experiment/ge_app/.direnv/python-3.7.2/lib/python3.7/site-packages/great_expectations/cli/suite.py", line 318, in suite_new
    raise e
  File "/home/shahin/Work/Experiment/ge_app/.direnv/python-3.7.2/lib/python3.7/site-packages/great_expectations/cli/suite.py", line 275, in suite_new
    open_docs=view,
  File "/home/shahin/Work/Experiment/ge_app/.direnv/python-3.7.2/lib/python3.7/site-packages/great_expectations/cli/datasource.py", line 906, in create_expectation_suite
    additional_batch_kwargs=additional_batch_kwargs)
  File "/home/shahin/Work/Experiment/ge_app/.direnv/python-3.7.2/lib/python3.7/site-packages/great_expectations/cli/datasource.py", line 833, in get_batch_kwargs
    batch_kwargs_generator_name=batch_kwargs_generator_name,
  File "/home/shahin/Work/Experiment/ge_app/.direnv/python-3.7.2/lib/python3.7/site-packages/great_expectations/cli/datasource.py", line 1107, in _get_batch_kwargs_from_generator_or_from_file_path
    batch = datasource.get_batch(batch_kwargs=batch_kwargs)
  File "/home/shahin/Work/Experiment/ge_app/.direnv/python-3.7.2/lib/python3.7/site-packages/great_expectations/datasource/pandas_datasource.py", line 205, in get_batch
    df, index=True).values).hexdigest()
  File "/home/shahin/Work/Experiment/ge_app/.direnv/python-3.7.2/lib/python3.7/site-packages/pandas/core/util/hashing.py", line 137, in hash_pandas_object
    h = _combine_hash_arrays(hashes, num_items)
  File "/home/shahin/Work/Experiment/ge_app/.direnv/python-3.7.2/lib/python3.7/site-packages/pandas/core/util/hashing.py", line 40, in _combine_hash_arrays
    first = next(arrays)
  File "/home/shahin/Work/Experiment/ge_app/.direnv/python-3.7.2/lib/python3.7/site-packages/pandas/core/util/hashing.py", line 136, in <genexpr>
    hashes = (x for x in _hashes)
  File "/home/shahin/Work/Experiment/ge_app/.direnv/python-3.7.2/lib/python3.7/site-packages/pandas/core/util/hashing.py", line 119, in <genexpr>
    hashes = (hash_array(series.values) for _, series in obj.items())
  File "/home/shahin/Work/Experiment/ge_app/.direnv/python-3.7.2/lib/python3.7/site-packages/pandas/core/util/hashing.py", line 306, in hash_array
    codes, categories = factorize(vals, sort=False)
  File "/home/shahin/Work/Experiment/ge_app/.direnv/python-3.7.2/lib/python3.7/site-packages/pandas/core/algorithms.py", line 636, in factorize
    values, na_sentinel=na_sentinel, size_hint=size_hint, na_value=na_value
  File "/home/shahin/Work/Experiment/ge_app/.direnv/python-3.7.2/lib/python3.7/site-packages/pandas/core/algorithms.py", line 484, in _factorize_array
    uniques, codes = table.factorize(values, na_sentinel=na_sentinel, na_value=na_value)
  File "pandas/_libs/hashtable_class_helper.pxi", line 1815, in pandas._libs.hashtable.PyObjectHashTable.factorize
  File "pandas/_libs/hashtable_class_helper.pxi", line 1731, in pandas._libs.hashtable.PyObjectHashTable._unique
TypeError: unhashable type: 'dict'
```

Of course, if you use a non-compressed file, it'll show a more polite version of the same error:

```
Cannot load file.
  - Please check the file and try again or select a different data file.
  - Error: unhashable type: 'dict'
```

## How to reproduce

Create a JSON file with the following content and try to create a suite out of it:

```json
[
    {"key": {"type": 2}}
]
```

## Resolution

As it's visible in the exception error, it's happening on [this call to](https://github.com/great-expectations/great_expectations/blob/develop/great_expectations/datasource/pandas_datasource.py#L205) `pandas.util.hash_pandas_object`. And that happens when the data frame size is below a fixed [threshold](https://github.com/great-expectations/great_expectations/blob/ff03bb0c8bf91ffe4d7f21108d6d4a6b4a3b165f/great_expectations/datasource/pandas_datasource.py#L27). 


Discussion:
As a hotfix test, I just commented out that `if` condition and the suite creation was successful :thinking:. However I'm not sure if that would cause issues in other places. 

Resolving patch:
diff --git a/great_expectations/datasource/pandas_datasource.py b/great_expectations/datasource/pandas_datasource.py
--- a/great_expectations/datasource/pandas_datasource.py
+++ b/great_expectations/datasource/pandas_datasource.py
@@ -1,6 +1,5 @@
 import datetime
 import uuid
-import hashlib
 import logging
 from functools import partial
 
@@ -20,7 +19,7 @@
 from great_expectations.core.batch import Batch
 from great_expectations.types import ClassConfig
 from great_expectations.exceptions import BatchKwargsError
-from .util import S3Url
+from .util import S3Url, hash_pandas_dataframe
 
 logger = logging.getLogger(__name__)
 
@@ -201,8 +200,7 @@ def get_batch(self, batch_kwargs, batch_parameters=None):
                                    batch_kwargs)
 
         if df.memory_usage().sum() < HASH_THRESHOLD:
-            batch_markers["pandas_data_fingerprint"] = hashlib.md5(pd.util.hash_pandas_object(
-                df, index=True).values).hexdigest()
+            batch_markers["pandas_data_fingerprint"] = hash_pandas_dataframe(df)
 
         return Batch(
             datasource_name=self.name,
diff --git a/great_expectations/datasource/util.py b/great_expectations/datasource/util.py
--- a/great_expectations/datasource/util.py
+++ b/great_expectations/datasource/util.py
@@ -1,4 +1,7 @@
+import pickle
+import hashlib
 from urllib.parse import urlparse
+import pandas as pd
 
 # S3Url class courtesy: https://stackoverflow.com/questions/42641315/s3-urls-get-bucket-name-and-path
 class S3Url(object):
@@ -43,3 +46,13 @@ def key(self):
     @property
     def url(self):
         return self._parsed.geturl()
+
+
+def hash_pandas_dataframe(df):
+    try:
+        obj = pd.util.hash_pandas_object(df, index=True).values
+    except TypeError:
+        # In case of facing unhashable objects (like dict), use pickle
+        obj = pickle.dumps(df, pickle.HIGHEST_PROTOCOL)
+
+    return hashlib.md5(obj).hexdigest()


Test patch:
diff --git a/tests/datasource/test_util.py b/tests/datasource/test_util.py
new file mode 100644
--- /dev/null
+++ b/tests/datasource/test_util.py
@@ -0,0 +1,17 @@
+import pandas as pd
+
+from great_expectations.datasource.util import hash_pandas_dataframe
+
+
+def test_hash_pandas_dataframe_hashable_df():
+    data = [{"col_1": 1}]
+    df1 = pd.DataFrame(data)
+    df2 = pd.DataFrame(data)
+    assert hash_pandas_dataframe(df1) == hash_pandas_dataframe(df2)
+
+
+def test_hash_pandas_dataframe_unhashable_df():
+    data = [{"col_1": {"val": 1}}]
+    df1 = pd.DataFrame(data)
+    df2 = pd.DataFrame(data)
+    assert hash_pandas_dataframe(df1) == hash_pandas_dataframe(df2)