Repository: pandas-dev/pandas
Pull request: TST: add read_json test #GH32383
- [x] closes #32383
- [x] tests added / passed
- [x] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Problem statement:
pd.read_json() leads to a segmentation fault
#### Code Sample

The following code leads to either a segmentation fault or a bus error:
```python
def load_dataframe(name):
    ...
    df_json = db.dataframes.find_one({"_id": name})["df"]
    df = pd.read_json(df_json)
    return df
```

But the following workaround fixes it:
```python
def load_dataframe(name):
    ...
    df_json = db.dataframes.find_one({"_id": name})["df"]
    df = pd.DataFrame().from_dict(json.loads(df_json))
    return df
```

#### Problem description
I am storing several jsonified dataframes in MongoDB. Trying to go straight from the json to the dataframe leads to either a segmentation fault or a bus error. I am storying several dataframes, but confusingly it only happens with one dataframe and not the others. The problem dataframe is identical in nature to other dataframes (same column names and data types). Unfortunately, it's not shareable but happy to answer any questions I can.

#### Output of ``pd.show_versions()``
 Note: I am using the latest pandas, 1.0.1. I downgraded to pandas 0.25.3 and had the same issue.

<details>
INSTALLED VERSIONS
------------------
commit           : None
python           : 3.7.0.final.0
python-bits      : 64
OS               : Darwin
OS-release       : 17.7.0
machine          : x86_64
processor        : i386
byteorder        : little
LC_ALL           : None
LANG             : en_US.UTF-8
LOCALE           : en_US.UTF-8

pandas           : 1.0.1
numpy            : 1.18.1
pytz             : 2019.3
dateutil         : 2.8.1
pip              : 20.0.2
setuptools       : 45.1.0
Cython           : None
pytest           : None
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : None
html5lib         : None
pymysql          : None
psycopg2         : None
jinja2           : 2.11.1
IPython          : 7.11.1
pandas_datareader: None
bs4              : None
bottleneck       : None
fastparquet      : None
gcsfs            : None
lxml.etree       : None
matplotlib       : 3.1.3
numexpr          : None
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : None
pytables         : None
pytest           : None
pyxlsb           : None
s3fs             : None
scipy            : 1.4.1
sqlalchemy       : None
tables           : None
tabulate         : None
xarray           : None
xlrd             : None
xlwt             : None
xlsxwriter       : None
numba            : None
</details>

Hints from issue discussion:
Can you post a copy/paste-able example (https://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports)?
Yes I can! Apologies for not doing so sooner. Thanks for the helpful MCVE post.

This leads to a segmentation fault on my machine:
```python
import pandas as pd

df = pd.DataFrame(
        {'_id': {'0': '0'},
        'category': {'0': 'Goods'},
        'recommender_id': {'0': '3'},
        'recommender_name_jp': {'0': '浦田'},
        'recommender_name_en': {'0': 'Urata'},
        'name_jp': {'0': '博多人形（松尾吉将まつお よしまさ）'},
        'name_en': {'0': 'Hakata Dolls Matsuo'}
        })

df_json = df.to_json()
pd.read_json(df_json)
```

But this does not:
```python 
import json
import pandas as pd

df = pd.DataFrame(
        {'_id': {'0': '0'},
        'category': {'0': 'Goods'},
        'recommender_id': {'0': '3'},
        'recommender_name_jp': {'0': '浦田'},
        'recommender_name_en': {'0': 'Urata'},
        'name_jp': {'0': '博多人形（松尾吉将まつお よしまさ）'},
        'name_en': {'0': 'Hakata Dolls Matsuo'}
        })

df_json = df.to_json()
df_dict = json.loads(df_json)
pd.DataFrame().from_dict(df_dict)
```

And, unfortunately, there is no helpful traceback. All I get is a `Segmentation fault: 11` or sometimes a `Bus error: 10`.
Hmm i cant reproduce this locally, so let's try to narrow down the example.  Do you need all of the entries in that dict to get the segfault, or can you remove some?  (id speculate that the ASCII entries can be removed)
I also can't reproduce using the code in https://github.com/pandas-dev/pandas/issues/32383#issuecomment-593252014 (using pandas master).
@gambingo can you try on pandas master?
Cant produce this either on master with OSX. Something possible might have been fixed on master in the meantime. Suppose it could use a test
cannot reproduce this for pandas master and v1.0.1 and v0.25.3 in Windows environment.

Commit message: remove expected df

Patch to review:
diff --git a/pandas/tests/io/json/test_json_table_schema.py b/pandas/tests/io/json/test_json_table_schema.py
index c0d40048a72fe..aa6ea9e080e16 100644
--- a/pandas/tests/io/json/test_json_table_schema.py
+++ b/pandas/tests/io/json/test_json_table_schema.py
@@ -237,6 +237,24 @@ def test_build_series(self):
 
         assert result == expected
 
+    def test_read_json_from_to_json_results(self):
+        # GH 32383
+        df = pd.DataFrame(
+            {
+                "_id": {"0": "0"},
+                "category": {"0": "Goods"},
+                "recommender_id": {"0": "3"},
+                "recommender_name_jp": {"0": "浦田"},
+                "recommender_name_en": {"0": "Urata"},
+                "name_jp": {"0": "博多人形（松尾吉将まつお よしまさ）"},
+                "name_en": {"0": "Hakata Dolls Matsuo"},
+            }
+        )
+        df_json = df.to_json()
+        result = pd.read_json(df_json)
+        result.index = result.index.astype(str)
+        tm.assert_frame_equal(result, df)
+
     def test_to_json(self):
         df = self.df.copy()
         df.index.name = "idx"


Code review comments:
diff_hunk: @@ -237,6 +237,24 @@ def test_build_series(self):
 
         assert result == expected
 
+    def test_read_json_from_to_json_results(self):
+        # GH 32383
+        df = pd.DataFrame(
+            {
+                "_id": {"0": "0"},
+                "category": {"0": "Goods"},
+                "recommender_id": {"0": "3"},
+                "recommender_name_jp": {"0": "浦田"},
+                "recommender_name_en": {"0": "Urata"},
+                "name_jp": {"0": "博多人形（松尾吉将まつお よしまさ）"},
+                "name_en": {"0": "Hakata Dolls Matsuo"},
+            }
+        )
+        df_json = df.to_json()
+        result = pd.read_json(df_json)
path: pandas/tests/io/json/test_json_table_schema.py
text: @user1:
if the test is the roundtrip, we could indicate this in the result = statement
```suggestion
        result = pd.read_json(df.to_json())
```