TST: Add test to ensure DF describe does not throw an error (#32409)
State: closed

- [x] closes #32409
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Notes:
- Tested using `pytest pandas/tests/frame/methods/test_describe.py::TestDataFrameDescribe::test_describe_does_not_raise_error` and tested error format by calling a raise error line in the try block `raise TypeError("Test message")`

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
This is very strange, but I  don't know that this test is actually covering the original issue. With pandas 1.0.1

```python
In [6]: >>> df = pd.DataFrame([{"test": {"a": "1"}}, {"test": {"a": "2"}}])

In [7]: r = df.describe()
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.map_locations()

TypeError: unhashable type: 'dict'
Exception ignored in: 'pandas._libs.index.IndexEngine._call_map_locations'
Traceback (most recent call last):
  File "pandas/_libs/hashtable_class_helper.pxi", line 1652, in pandas._libs.hashtable.PyObjectHashTable.map_locations
TypeError: unhashable type: 'dict'

In [8]: r
Out[8]:
              test
count            2
unique           2
top     {'a': '1'}
freq             1
```

So the code executed. The `TypeError` printed there is apparently handled, but it's still printed out to the screen, and the traceback has been set.

@luckyvs1 can you verify that the test you added *fails* with pandas 1.0.1? You should be able just make a single `test_.py` file with your new test in it (make sure it's not in your `pandas` directory). And then create a new environment with pandas 1.0.1 and pytest and run the test?


Thank you for the feedback @TomAugspurger -- unfortunately I won't be able to look into this till this weekend, but will investigate if the test case written fails in the tagged `1.0.1` release version and if it doesn’t will modify it accordingly and retest it passes on the latest.
@TomAugspurger  I just had a chance to verify on the release `1.0.1` [branch](https://github.com/luckyvs1/pandas/tree/add-test-for-describe-v1.0.1) this test does indeed throw an error:

```
ubuntu@ubuntu-MacBookPro:~/Desktop/Github/pandas$ pytest pandas/tests/frame/methods/test_describe.py::TestDataFrameDescribe::test_describe_does_not_raise_error
================================================================= test session starts ==================================================================
platform linux -- Python 3.6.9, pytest-5.4.3, py-1.9.0, pluggy-0.13.1
rootdir: /home/ubuntu/Desktop/Github/pandas, inifile: setup.cfg
plugins: hypothesis-5.19.0, cov-2.10.0, responses-0.4.0
collected 1 item                                                                                                                                       

pandas/tests/frame/methods/test_describe.py F                                                                                                    [100%]

======================================================================= FAILURES =======================================================================
_______________________________________________ TestDataFrameDescribe.test_describe_does_not_raise_error _______________________________________________

self = <pandas.tests.frame.methods.test_describe.TestDataFrameDescribe object at 0x7f28f1cf51d0>

    def test_describe_does_not_raise_error(self):
        # GH#32409
        df = pd.DataFrame([{"test": {"a": "1"}}, {"test": {"a": "2"}}])
        expected = DataFrame(
            {"test": [2, 2, {"a": "1"}, 1]}, index=["count", "unique", "top", "freq"]
        )
>       result = df.describe()

pandas/tests/frame/methods/test_describe.py:340: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
pandas/core/generic.py:9940: in describe
    ldesc = [describe_1d(s) for _, s in data.items()]
pandas/core/generic.py:9940: in <listcomp>
    ldesc = [describe_1d(s) for _, s in data.items()]
pandas/core/generic.py:9923: in describe_1d
    return describe_categorical_1d(data)
pandas/core/generic.py:9879: in describe_categorical_1d
    objcounts = data.value_counts()
pandas/core/base.py:1244: in value_counts
    dropna=dropna,
pandas/core/algorithms.py:723: in value_counts
    keys, counts = _value_counts_arraylike(values, dropna)
pandas/core/algorithms.py:767: in _value_counts_arraylike
    keys, counts = f(values, dropna)
pandas/_libs/hashtable_func_helper.pxi:336: in pandas._libs.hashtable.value_count_object
    cpdef value_count_object(ndarray[object] values, bint dropna):
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   result_keys = np.empty(table.n_occupied, dtype=np.object)
E   TypeError: unhashable type: 'dict'

pandas/_libs/hashtable_func_helper.pxi:347: TypeError
=============================================================== short test summary info ================================================================
FAILED pandas/tests/frame/methods/test_describe.py::TestDataFrameDescribe::test_describe_does_not_raise_error - TypeError: unhashable type: 'dict'
================================================================== 1 failed in 0.39s ===================================================================
```

and the output of the test on my current PR branch which is built off release `1.0.5`:

```
ubuntu@ubuntu-MacBookPro:~/Desktop/Github/pandas$ pytest pandas/tests/frame/methods/test_describe.py::TestDataFrameDescribe::test_describe_does_not_raise_error
================================================================= test session starts ==================================================================
platform linux -- Python 3.6.9, pytest-5.4.3, py-1.9.0, pluggy-0.13.1
rootdir: /home/ubuntu/Desktop/Github/pandas, inifile: setup.cfg
plugins: hypothesis-5.19.0, cov-2.10.0, responses-0.4.0
collected 1 item                                                                                                                                       

pandas/tests/frame/methods/test_describe.py .                                                                                                    [100%]

================================================================== 1 passed in 0.04s ===================================================================
```
something weird going on. On windows with pandas 1.0.1, I get

```
$ pytest ~/test.py -vv -s
============================= test session starts =============================
platform win32 -- Python 3.8.3, pytest-5.4.3, py-1.9.0, pluggy-0.13.1 -- C:\Users\simon\Anaconda3\envs\pandas-1.0.1\python.exe
cachedir: .pytest_cache
rootdir: C:\Users\simon
collecting ... collected 1 item

..\test.py::test_describe_does_not_raise_error Traceback (most recent call last):
  File "pandas\_libs\hashtable_class_helper.pxi", line 1652, in pandas._libs.hashtable.PyObjectHashTable.map_locations
TypeError: unhashable type: 'dict'
Exception ignored in: 'pandas._libs.index.IndexEngine._call_map_locations'
Traceback (most recent call last):
  File "pandas\_libs\hashtable_class_helper.pxi", line 1652, in pandas._libs.hashtable.PyObjectHashTable.map_locations
TypeError: unhashable type: 'dict'
PASSED

============================== 1 passed in 0.79s ==============================
```
@simonjayhawkins  Using `Python 3.6.9` with `Pandas 1.0.1` on `Linux Ubuntu 18.04.4 LTS` this issue doesn't seem to be resolved:

I imported `pandas 1.0.1` in the Python shell and performed the example as described in the issue #32409 and the describe command indeed did not run as seen in the result:

## Reproduced example with failed describe in shell with `pandas 1.0.1`
```
ubuntu@ubuntu-MacBookPro:~/Desktop/Github$ python3
Python 3.6.9 (default, Apr 18 2020, 01:56:04) 
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas as pd
>>> import numpy as np
>>> print(pd.__version__)
1.0.1
>>> df = pd.DataFrame([{"test": {"a": "1"}}, {"test": {"a": "2"}}])
>>> print(df)
         test
0  {'a': '1'}
1  {'a': '2'}
>>> print(df.describe())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ubuntu/.local/lib/python3.6/site-packages/pandas/core/generic.py", line 9940, in describe
    ldesc = [describe_1d(s) for _, s in data.items()]
  File "/home/ubuntu/.local/lib/python3.6/site-packages/pandas/core/generic.py", line 9940, in <listcomp>
    ldesc = [describe_1d(s) for _, s in data.items()]
  File "/home/ubuntu/.local/lib/python3.6/site-packages/pandas/core/generic.py", line 9923, in describe_1d
    return describe_categorical_1d(data)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/pandas/core/generic.py", line 9879, in describe_categorical_1d
    objcounts = data.value_counts()
  File "/home/ubuntu/.local/lib/python3.6/site-packages/pandas/core/base.py", line 1244, in value_counts
    dropna=dropna,
  File "/home/ubuntu/.local/lib/python3.6/site-packages/pandas/core/algorithms.py", line 723, in value_counts
    keys, counts = _value_counts_arraylike(values, dropna)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/pandas/core/algorithms.py", line 767, in _value_counts_arraylike
    keys, counts = f(values, dropna)
  File "pandas/_libs/hashtable_func_helper.pxi", line 336, in pandas._libs.hashtable.value_count_object
  File "pandas/_libs/hashtable_func_helper.pxi", line 347, in pandas._libs.hashtable.value_count_object
TypeError: unhashable type: 'dict'
>>> result = df.describe()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ubuntu/.local/lib/python3.6/site-packages/pandas/core/generic.py", line 9940, in describe
    ldesc = [describe_1d(s) for _, s in data.items()]
  File "/home/ubuntu/.local/lib/python3.6/site-packages/pandas/core/generic.py", line 9940, in <listcomp>
    ldesc = [describe_1d(s) for _, s in data.items()]
  File "/home/ubuntu/.local/lib/python3.6/site-packages/pandas/core/generic.py", line 9923, in describe_1d
    return describe_categorical_1d(data)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/pandas/core/generic.py", line 9879, in describe_categorical_1d
    objcounts = data.value_counts()
  File "/home/ubuntu/.local/lib/python3.6/site-packages/pandas/core/base.py", line 1244, in value_counts
    dropna=dropna,
  File "/home/ubuntu/.local/lib/python3.6/site-packages/pandas/core/algorithms.py", line 723, in value_counts
    keys, counts = _value_counts_arraylike(values, dropna)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/pandas/core/algorithms.py", line 767, in _value_counts_arraylike
    keys, counts = f(values, dropna)
  File "pandas/_libs/hashtable_func_helper.pxi", line 336, in pandas._libs.hashtable.value_count_object
  File "pandas/_libs/hashtable_func_helper.pxi", line 347, in pandas._libs.hashtable.value_count_object
TypeError: unhashable type: 'dict'
>>> print(result)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'result' is not defined
```

## Ubuntu Version
```
ubuntu@ubuntu-MacBookPro:~/Desktop/Github$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.4 LTS
Release:	18.04
Codename:	bionic
```

## Output of pd.show_versions()
```
>>> pd.show_versions()

INSTALLED VERSIONS
------------------
commit           : None
python           : 3.6.9.final.0
python-bits      : 64
OS               : Linux
OS-release       : 4.15.0-111-generic
machine          : x86_64
processor        : x86_64
byteorder        : little
LC_ALL           : None
LANG             : en_US.UTF-8
LOCALE           : en_US.UTF-8

pandas           : 1.0.1
numpy            : 1.19.0
pytz             : 2020.1
dateutil         : 2.8.1
pip              : 9.0.1
setuptools       : 49.2.0
Cython           : 0.29.21
pytest           : 5.4.3
hypothesis       : 5.19.0
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : 4.4.2
html5lib         : 0.999999999
pymysql          : None
psycopg2         : None
jinja2           : 2.10.3
IPython          : 7.10.1
pandas_datareader: None
bs4              : None
bottleneck       : None
fastparquet      : None
gcsfs            : None
lxml.etree       : 4.4.2
matplotlib       : None
numexpr          : None
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : None
pytables         : None
pytest           : 5.4.3
pyxlsb           : None
s3fs             : None
scipy            : 1.5.1
sqlalchemy       : None
tables           : None
tabulate         : None
xarray           : None
xlrd             : None
xlwt             : None
xlsxwriter       : None
numba            : None
```

Please let me know if you need any additional information or if I should create a new issue for this or update the issue #32409  with a comment.
I'm not sure how this should be tested (or whether it can be tested). my concern is that the original cause of the issue of not catching exceptions in the cython code is not solved (see https://github.com/pandas-dev/pandas/pull/34113#issue-416019180). I'm not sure how the issue is fixed on master. Maybe I did the git bisect wrong in https://github.com/pandas-dev/pandas/issues/32409#issuecomment-606627286. The pr referenced there changed is_scalar and is_iterator, so if that PR did fix the describe issue, it must have changed the behaviour of one of those functions. Either way, I think there is a problem somewhere.
For reference I'm also able to reproduce the issue described in #32409 on my Mac as well using `Pandas 1.0.1`.

## Reproduced example with failed describe in shell with `pandas 1.0.1`
```
Luckys-MacBook-Pro:~ lucky$ python3
Python 3.6.5 | packaged by conda-forge | (default, Apr  6 2018, 13:44:09) 
[GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas as pd
>>> import numpy as np
>>> print(pd.__version__)
1.0.1
>>> df = pd.DataFrame([{"test": {"a": "1"}}, {"test": {"a": "2"}}])
>>> print(df)
         test
0  {'a': '1'}
1  {'a': '2'}
>>> print(df.describe())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/lucky/anaconda/lib/python3.6/site-packages/pandas/core/generic.py", line 9940, in describe
    ldesc = [describe_1d(s) for _, s in data.items()]
  File "/Users/lucky/anaconda/lib/python3.6/site-packages/pandas/core/generic.py", line 9940, in <listcomp>
    ldesc = [describe_1d(s) for _, s in data.items()]
  File "/Users/lucky/anaconda/lib/python3.6/site-packages/pandas/core/generic.py", line 9923, in describe_1d
    return describe_categorical_1d(data)
  File "/Users/lucky/anaconda/lib/python3.6/site-packages/pandas/core/generic.py", line 9879, in describe_categorical_1d
    objcounts = data.value_counts()
  File "/Users/lucky/anaconda/lib/python3.6/site-packages/pandas/core/base.py", line 1244, in value_counts
    dropna=dropna,
  File "/Users/lucky/anaconda/lib/python3.6/site-packages/pandas/core/algorithms.py", line 723, in value_counts
    keys, counts = _value_counts_arraylike(values, dropna)
  File "/Users/lucky/anaconda/lib/python3.6/site-packages/pandas/core/algorithms.py", line 767, in _value_counts_arraylike
    keys, counts = f(values, dropna)
  File "pandas/_libs/hashtable_func_helper.pxi", line 336, in pandas._libs.hashtable.value_count_object
  File "pandas/_libs/hashtable_func_helper.pxi", line 347, in pandas._libs.hashtable.value_count_object
TypeError: unhashable type: 'dict'
>>> result = df.describe()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/lucky/anaconda/lib/python3.6/site-packages/pandas/core/generic.py", line 9940, in describe
    ldesc = [describe_1d(s) for _, s in data.items()]
  File "/Users/lucky/anaconda/lib/python3.6/site-packages/pandas/core/generic.py", line 9940, in <listcomp>
    ldesc = [describe_1d(s) for _, s in data.items()]
  File "/Users/lucky/anaconda/lib/python3.6/site-packages/pandas/core/generic.py", line 9923, in describe_1d
    return describe_categorical_1d(data)
  File "/Users/lucky/anaconda/lib/python3.6/site-packages/pandas/core/generic.py", line 9879, in describe_categorical_1d
    objcounts = data.value_counts()
  File "/Users/lucky/anaconda/lib/python3.6/site-packages/pandas/core/base.py", line 1244, in value_counts
    dropna=dropna,
  File "/Users/lucky/anaconda/lib/python3.6/site-packages/pandas/core/algorithms.py", line 723, in value_counts
    keys, counts = _value_counts_arraylike(values, dropna)
  File "/Users/lucky/anaconda/lib/python3.6/site-packages/pandas/core/algorithms.py", line 767, in _value_counts_arraylike
    keys, counts = f(values, dropna)
  File "pandas/_libs/hashtable_func_helper.pxi", line 336, in pandas._libs.hashtable.value_count_object
  File "pandas/_libs/hashtable_func_helper.pxi", line 347, in pandas._libs.hashtable.value_count_object
TypeError: unhashable type: 'dict'
>>> print(result)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'result' is not defined
```

## Mac Version
```
Luckys-MacBook-Pro:~ lucky$ sw_vers
ProductName:	Mac OS X
ProductVersion:	10.15.4
BuildVersion:	19E287
```

## Output of pd.show_versions()
```
>>> pd.show_versions()

INSTALLED VERSIONS
------------------
commit           : None
python           : 3.6.5.final.0
python-bits      : 64
OS               : Darwin
OS-release       : 19.4.0
machine          : x86_64
processor        : i386
byteorder        : little
LC_ALL           : None
LANG             : en_CA.UTF-8
LOCALE           : en_CA.UTF-8

pandas           : 1.0.1
numpy            : 1.17.5
pytz             : 2017.2
dateutil         : 2.8.1
pip              : 20.1.1
setuptools       : 49.1.0.post20200704
Cython           : 0.25.2
pytest           : 5.4.3
hypothesis       : 5.19.0
sphinx           : 1.5.6
blosc            : None
feather          : None
xlsxwriter       : 0.9.6
lxml.etree       : 3.7.3
html5lib         : 0.999
pymysql          : None
psycopg2         : None
jinja2           : 2.9.6
IPython          : 5.3.0
pandas_datareader: None
bs4              : 4.6.0
bottleneck       : 1.3.2
fastparquet      : None
gcsfs            : None
lxml.etree       : 3.7.3
matplotlib       : 3.2.2
numexpr          : 2.7.1
odfpy            : None
openpyxl         : 2.4.7
pandas_gbq       : None
pyarrow          : None
pytables         : None
pytest           : 5.4.3
pyxlsb           : None
s3fs             : None
scipy            : 1.5.0
sqlalchemy       : 1.1.9
tables           : 3.6.1
tabulate         : None
xarray           : None
xlrd             : 1.0.0
xlwt             : 1.2.0
xlsxwriter       : 0.9.6
numba            : 0.49.1
```
If this needs to be fixed in 1.0.1.x then this test or similar can be added to that release and check if the added test passes on Mac, Linux versions supported in addition to Windows. Since the current behaviour seems to be it's working on Windows, but not Mac and Linux (at least the versions I provided).

In the versions where the exception is handled while still being outputted like the current Windows 1.0.1, I’m also not sure at the moment on how that can be tested or how to currently test the lack of exception output as seen in latest 1.0.5+ versions.

Edit: Maybe a test that writes the output to a file and check if the contents doesn't contain references to `TypeError: unhashable type: 'dict'` to check the exception is not being thrown even if the functionality is working and Pytest capturing may help haven’t tested it yet: https://docs.pytest.org/en/latest/capture.html
@luckyvs1 this got lost, can you merge master and ping on green
Closing in favor of https://github.com/pandas-dev/pandas/pull/38403
Note for reference: Reason the diff turned out to be the way it is was due to the forked branch not being up to date with the latest, to fix this at this point do the following with the rebase applied can do the following:

1. `git flog` to find the commit before the rebase
2. `git reset --hard HEAD@{#}` to reset the branch
3. `git push --force` force update to fix the commit diff

From here, we can update the forked master first then rebase off that branch:
1. While on the master branch on the forked repo use the following to update the forked master branch: https://stackoverflow.com/questions/7244321/how-do-i-update-a-github-forked-repository
2. Then switch over to the branch `add-test-for-describe` and rebase with the forked master and the resolve any conflicts as needed the commit history will be a clean rebase as intended