Repository: pandas-dev/pandas
Pull request: BUG: Integer Overflow in read_json with big number in string
- [x] closes #30320
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Problem statement:
Read_json overflow error when json contains big number strings 
#### Code Sample, a copy-pastable example if possible

```python
import json
import pandas as pd

test_data = [{"col": "31900441201190696999"}, {"col": "Text"}]
test_json = json.dumps(test_data)
pd.read_json(test_json)

```
#### Problem description

The current behaviour doesn't return a dateframe for a valid JSON. Note when the number is smaller, it works fine. It also works when only big numbers are present. It would be cool to have it work with big numbers as it works for small numbers.

#### Expected Output
A dataframe with a number and string
```
       col
0  3.190044e+19
1     Text
```

#### Output of ``pd.read_json()``

<details>
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".../.venv/lib/python3.6/site-packages/pandas/io/json/_json.py", line 592, in read_json
    result = json_reader.read()
  File ".../.venv/lib/python3.6/site-packages/pandas/io/json/_json.py", line 717, in read
    obj = self._get_object_parser(self.data)
  File ".../.venv/lib/python3.6/site-packages/pandas/io/json/_json.py", line 739, in _get_object_parser
    obj = FrameParser(json, **kwargs).parse()
  File ".../.venv/lib/python3.6/site-packages/pandas/io/json/_json.py", line 855, in parse
    self._try_convert_types()
  File ".../.venv/lib/python3.6/site-packages/pandas/io/json/_json.py", line 1151, in _try_convert_types
    lambda col, c: self._try_convert_data(col, c, convert_dates=False)
  File ".../.venv/lib/python3.6/site-packages/pandas/io/json/_json.py", line 1131, in _process_converter
    new_data, result = f(col, c)
  File ".../.venv/lib/python3.6/site-packages/pandas/io/json/_json.py", line 1151, in <lambda>
    lambda col, c: self._try_convert_data(col, c, convert_dates=False)
  File ".../.venv/lib/python3.6/site-packages/pandas/io/json/_json.py", line 927, in _try_convert_data
    new_data = data.astype("int64")
  File ".../.venv/lib/python3.6/site-packages/pandas/core/generic.py", line 5882, in astype
    dtype=dtype, copy=copy, errors=errors, **kwargs
  File ".../.venv/lib/python3.6/site-packages/pandas/core/internals/managers.py", line 581, in astype
    return self.apply("astype", dtype=dtype, **kwargs)
  File ".../.venv/lib/python3.6/site-packages/pandas/core/internals/managers.py", line 438, in apply
    applied = getattr(b, f)(**kwargs)
  File ".../.venv/lib/python3.6/site-packages/pandas/core/internals/blocks.py", line 559, in astype
    return self._astype(dtype, copy=copy, errors=errors, values=values, **kwargs)
  File ".../.venv/lib/python3.6/site-packages/pandas/core/internals/blocks.py", line 643, in _astype
    values = astype_nansafe(vals1d, dtype, copy=True, **kwargs)
  File ".../.venv/lib/python3.6/site-packages/pandas/core/dtypes/cast.py", line 707, in astype_nansafe
    return lib.astype_intsafe(arr.ravel(), dtype).reshape(arr.shape)
  File "pandas/_libs/lib.pyx", line 547, in pandas._libs.lib.astype_intsafe
OverflowError: Python int too large to convert to C long

</details>

Hints from issue discussion:
take
I'm new to Open Source contributions so please bear with me. It seems that we are coercing ints wherever possible while parsing JSON. The code b/w line 943 to line 950 in file [pandas/io/json/_json.py](https://github.com/pandas-dev/pandas/blob/4e807a2923804eb231eb9b7c991273e860c25726/pandas/io/json/_json.py#L943) is what is causing the problem. The int coerce is being checked using a try except which only catches TypeErrors and ValueErrors. If it catches an OverflowError too then things work as intended. Will submit a PR regarding this soon.

Commit message: BUG: Integer Overflow in read_json with big number in string

Patch to review:
diff --git a/doc/source/whatsnew/v1.0.0.rst b/doc/source/whatsnew/v1.0.0.rst
index e44fec112c5c1..6e0fc2b3ecece 100755
--- a/doc/source/whatsnew/v1.0.0.rst
+++ b/doc/source/whatsnew/v1.0.0.rst
@@ -803,6 +803,7 @@ I/O
 - Bug in :func:`read_json` where default encoding was not set to ``utf-8`` (:issue:`29565`)
 - Bug in :class:`PythonParser` where str and bytes were being mixed when dealing with the decimal field (:issue:`29650`)
 - :meth:`read_gbq` now accepts ``progress_bar_type`` to display progress bar while the data downloads. (:issue:`29857`)
+- Bug in :meth: `read_json` where integer overflow was occuring when json contains big number strings. (:issue:`30320`)
 
 Plotting
 ^^^^^^^^
diff --git a/pandas/io/json/_json.py b/pandas/io/json/_json.py
index 30c1c2d59e983..d7abf921e69fc 100644
--- a/pandas/io/json/_json.py
+++ b/pandas/io/json/_json.py
@@ -946,7 +946,7 @@ def _try_convert_data(self, name, data, use_dtypes=True, convert_dates=True):
                 if (new_data == data).all():
                     data = new_data
                     result = True
-            except (TypeError, ValueError):
+            except (TypeError, ValueError, OverflowError):
                 pass
 
         # coerce ints to 64
diff --git a/pandas/tests/io/json/test_pandas.py b/pandas/tests/io/json/test_pandas.py
index bce3d1de849aa..efeb6fd008798 100644
--- a/pandas/tests/io/json/test_pandas.py
+++ b/pandas/tests/io/json/test_pandas.py
@@ -1601,3 +1601,16 @@ def test_json_indent_all_orients(self, orient, expected):
     def test_json_negative_indent_raises(self):
         with pytest.raises(ValueError, match="must be a nonnegative integer"):
             pd.DataFrame().to_json(indent=-1)
+
+    @pytest.mark.parametrize(
+        "json,expected",
+        [
+            (
+                json.dumps([{"col": "31900441201190696999"}, {"col": "3190044"}]),
+                DataFrame({"col": [31900441201190696999.0, 3190044.0]}),
+            ),
+        ],
+    )
+    def test_frame_int_overflow(self, json, expected):
+        result = read_json(json)
+        tm.assert_frame_equal(result, expected)


Code review comments:
diff_hunk: @@ -1601,3 +1601,16 @@ def test_json_indent_all_orients(self, orient, expected):
     def test_json_negative_indent_raises(self):
         with pytest.raises(ValueError, match="must be a nonnegative integer"):
             pd.DataFrame().to_json(indent=-1)
+
+    @pytest.mark.parametrize(
+        "json,expected",
+        [
+            (
+                json.dumps([{"col": "31900441201190696999"}, {"col": "3190044"}]),
+                DataFrame({"col": [31900441201190696999.0, 3190044.0]}),
+            ),
+        ],
+    )
+    def test_frame_int_overflow(self, json, expected):
+        result = read_json(json)
path: pandas/tests/io/json/test_pandas.py
text: @user1:
Can you add the GH issue number as a comment here?

@author:
Sure, Issue no. #30320

@user1:
Can you add this to the code? Should see examples on other tests

diff_hunk: @@ -1601,3 +1601,16 @@ def test_json_indent_all_orients(self, orient, expected):
     def test_json_negative_indent_raises(self):
         with pytest.raises(ValueError, match="must be a nonnegative integer"):
             pd.DataFrame().to_json(indent=-1)
+
+    @pytest.mark.parametrize(
path: pandas/tests/io/json/test_pandas.py
text: @user1:
Since you only have one pair of values shouldn't need to parametrize at all; just add to body of test

@author:
True, I initially thought of going with multiple test cases. Should I rewrite it?

@author:
Removed parametrize