Repository: pandas-dev/pandas

Issue:
pandas.DataFrame.replace seems taking number string as integer and run into overflow error
#### Code Sample

```python
import pandas as pd

x = {'user_id':['100000715097692381911', 
                '100003840837471130074'], 
     'item_id': [1, 2]
     }
dfx = pd.DataFrame(x)
dfx['user_id'].replace(
    {
     '100000715097692381911': 0, 
     '100003840837471130074': 1
     }, inplace=True)
```
#### Problem description
I got overflow error:

File "/home/grain/ml/lib/python3.6/site-packages/pandas/core/dtypes/cast.py", line 683, in astype_nansafe
    return lib.astype_intsafe(arr.ravel(), dtype).reshape(arr.shape)
  File "pandas/_libs/lib.pyx", line 546, in pandas._libs.lib.astype_intsafe
OverflowError: Python int too large to convert to C long

*This error occurs on my laptop with pandas 0.24.1, python 3.6.7. The example code can handle the replacement correctly on another computer with an older version pandas 0.20.3.*

#### Expected Output

Since I loaded big id numbers like '100000715097692381911' as string type, the pandas.DataFrame.replace() method should replace it with the corresponding value in the dictionary. But I got overflow error. It seemed pandas took the id string as integer?

And if I modify just the first element of 'user_id' column to 's100000715097692381911', then the code won't run into overflow.

#### Output of ``pd.show_versions()``

<details>

[paste the output of ``pd.show_versions()`` here below this line]
INSTALLED VERSIONS
------------------
commit: None
python: 3.6.7.final.0
python-bits: 64
OS: Linux
OS-release: 4.15.0-46-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.24.1
pytest: None
pip: 19.0.3
setuptools: 40.6.3
Cython: None
numpy: 1.16.0
scipy: 1.2.0
pyarrow: None
xarray: None
IPython: 7.2.0
sphinx: None
patsy: None
dateutil: 2.7.5
pytz: 2018.3
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: 3.0.2
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml.etree: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
gcsfs: None

</details>



Discussion:
Hmm ok that is strange - investigation and PRs would certainly be welcome
I'm new to working on pandas, so please note that some of the below may be incorrect.

I've looked into this a bit, and it looks like the cause is code added in #21477 in `Block._replace_coerce`, specifically the `coerce_to_target_dtype` function.

This didn't exist in the older version of pandas that the above code works on, and conversion was handled totally by the `ObjectBlock.convert` function.  At some point, it seems that function no longer met all the needs of converting, and `coerce_to_target_dtype` was introduced into the replace process to meet some of those needs.  However, that function tries too forcefully to convert the original Block (and it does so before it even finishes replacing values!), and thus we get an OverflowError in this case.

An easy fix to this is to just add `OverflowError ` to the list of errors `coerce_to_target_dtype` can catch ([here](https://github.com/pandas-dev/pandas/blob/master/pandas/core/internals/blocks.py#L1082)), which would fix this and retain other existing functionality.  However, I'm not sure if `coerce_to_target_dtype` is *meant* to fail in the case of an int that is too large.

If we can't change that, a much more narrow fix would be to only call `coerce_to_target_dtype` after the last replacement is done, but that doesn't some other cases that the old code could handle (for instance: replacing one of those values in the example instead of both of them).

I couldn't see a fix we could make to `ObjectBlock.convert` to handle all conversions without needing to call `coerce_to_target_dtype`.  I'm guessing this has to do with numpy changes, or perhaps changes to `maybe_convert_objects`, but I'm not sure.
@ArtificialQualia thanks a lot for your input!

> However, that function tries too forcefully to convert the original Block (and it does so before it even finishes replacing values!

I haven't stepped through the code myself but this seems to be the actual root issue - have you tried debugging this behavior to see if a fix can be applied to prevent this?
@WillAyd 

I did try both the fixes I mentioned in my comment.  They fix the issue mentioned in this issue, as well as passing all the tests in `tests\series\test_replace.py`.

If you want my recommendation, I would add `OverflowError` to `coerce_to_target_dtype` @ [line 1082](https://github.com/pandas-dev/pandas/blob/master/pandas/core/internals/blocks.py#L1082) and also move the call to `coerce_to_target_dtype` in `_replace_coerce` to after `putmask` so the coercion happens after the replacement.  Possibly also add in a `if convert:` around the `coerce_to_target_dtype` call so it only happens on the last entry, which is similar to how `ObjectBlock.convert` works which I'm assuming is for performance reasons.
@ArtificialQualia if you'd like to submit a PR would probably be easier to review and give you feedback with that. Thanks again!
No problem, I'll submit a PR.

Resolving patch:
diff --git a/pandas/core/internals/blocks.py b/pandas/core/internals/blocks.py
--- a/pandas/core/internals/blocks.py
+++ b/pandas/core/internals/blocks.py
@@ -1079,7 +1079,7 @@ def coerce_to_target_dtype(self, other):
 
         try:
             return self.astype(dtype)
-        except (ValueError, TypeError):
+        except (ValueError, TypeError, OverflowError):
             pass
 
         return self.astype(object)
@@ -3210,7 +3210,7 @@ def _putmask_smart(v, m, n):
                 nv = v.copy()
                 nv[m] = nn_at
                 return nv
-    except (ValueError, IndexError, TypeError):
+    except (ValueError, IndexError, TypeError, OverflowError):
         pass
 
     n = np.asarray(n)


Test patch:
diff --git a/pandas/tests/series/test_replace.py b/pandas/tests/series/test_replace.py
--- a/pandas/tests/series/test_replace.py
+++ b/pandas/tests/series/test_replace.py
@@ -280,3 +280,17 @@ def test_replace_mixed_types_with_string(self):
         result = s.replace([2, '4'], np.nan)
         expected = pd.Series([1, np.nan, 3, np.nan, 4, 5])
         tm.assert_series_equal(expected, result)
+
+    def test_replace_with_no_overflowerror(self):
+        # GH 25616
+        # casts to object without Exception from OverflowError
+        s = pd.Series([0, 1, 2, 3, 4])
+        result = s.replace([3], ['100000000000000000000'])
+        expected = pd.Series([0, 1, 2, '100000000000000000000', 4])
+        tm.assert_series_equal(result, expected)
+
+        s = pd.Series([0, '100000000000000000000',
+                      '100000000000000000001'])
+        result = s.replace(['100000000000000000000'], [1])
+        expected = pd.Series([0, 1, '100000000000000000001'])
+        tm.assert_series_equal(result, expected)