Repository: pandas-dev/pandas
Pull request: TST: added test for pd.where overflow error GH31687
Added a test for pd.where to /indexing/test_where.py

- [x] closes #31687 
- [x] tests added / passed
- [x] passes all pre-commit code checks
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Problem statement:
pd.where OverflowError with large numbers
#### Code Sample, a copy-pastable example if possible

```python
import pandas as pd
import numpy as np

df = pd.DataFrame([[1.0, 2e25],[np.nan, 0.1]])

# Works when applied to individual columns
print(df[0].where(pd.notnull(df[0]), None))
print(df[1].where(pd.notnull(df[1]), None))

# Breaks for whole dataframe
print(df.where(pd.notnull(df), None))

```
#### Problem description
The above code does not work with 1.0.0, but used to work with at least 0.25.0.
Replacing large floats with pd.where breaks 
Running pd.where on a dataframe that contains large float values and the replacement value is of a different dtype throws `OverflowError: int too big to convert`:

```
    applied = getattr(b, f)(**kwargs)
  File "***\venv\lib\site-packages\pandas\core\internals\blocks.py", line 1426, in where
    return self._maybe_downcast(blocks, "infer")
  File "***\venv\lib\site-packages\pandas\core\internals\blocks.py", line 514, in _maybe_downcast
    return _extend_blocks([b.downcast(downcast) for b in blocks])
  File "***\venv\lib\site-packages\pandas\core\internals\blocks.py", line 514, in <listcomp>
    return _extend_blocks([b.downcast(downcast) for b in blocks])
  File "***\venv\lib\site-packages\pandas\core\internals\blocks.py", line 552, in downcast
    return self.split_and_operate(None, f, False)
  File "***\venv\lib\site-packages\pandas\core\internals\blocks.py", line 496, in split_and_operate
    nv = f(m, v, i)
  File "***\venv\lib\site-packages\pandas\core\internals\blocks.py", line 549, in f
    val = maybe_downcast_to_dtype(val, dtype="infer")
  File "***\venv\lib\site-packages\pandas\core\dtypes\cast.py", line 135, in maybe_downcast_to_dtype
    converted = maybe_downcast_numeric(result, dtype, do_round)
  File "***\venv\lib\site-packages\pandas\core\dtypes\cast.py", line 222, in maybe_downcast_numeric
    new_result = trans(result).astype(dtype)
OverflowError: int too big to convert
```

Replacing data one column at a time works.


#### Expected Output
```
Name: 1, dtype: float64
      0      1
0     1  2e+25
1  None    0.1
```

#### Output of ``pd.show_versions()``

<details>

[paste the output of ``pd.show_versions()`` here below this line]
INSTALLED VERSIONS
------------------
commit           : None
python           : 3.7.3.final.0
python-bits      : 64
OS               : Windows
OS-release       : 10
machine          : AMD64
processor        : Intel64 Family 6 Model 142 Stepping 9, GenuineIntel
byteorder        : little
LC_ALL           : None
LANG             : None
LOCALE           : None.None
pandas           : 1.0.0
numpy            : 1.16.2
pytz             : 2018.9
dateutil         : 2.8.0
pip              : 10.0.1
setuptools       : 39.1.0
Cython           : None
pytest           : 4.4.0
hypothesis       : None
sphinx           : 2.0.0
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : 4.3.3
html5lib         : None
pymysql          : None
psycopg2         : None
jinja2           : 2.11.1
IPython          : None
pandas_datareader: None
bs4              : None
bottleneck       : None
fastparquet      : None
gcsfs            : None
lxml.etree       : 4.3.3
matplotlib       : None
numexpr          : None
odfpy            : None
openpyxl         : 2.6.2
pandas_gbq       : None
pyarrow          : None
pytables         : None
pytest           : 4.4.0
pyxlsb           : None
s3fs             : None
scipy            : None
sqlalchemy       : 1.3.3
tables           : None
tabulate         : None
xarray           : None
xlrd             : 1.2.0
xlwt             : None
xlsxwriter       : None
numba            : None
</details>

Hints from issue discussion:
Seems to be still broken on 1.0.1. 
**print(df.where(pd.notnull(df), 'some_str'))** raises the same exception.
However **print(df.where(pd.notnull(df), 1))** works.

Debugging through the code, it seems on this line **new_result = trans(result).astype(dtype)**, the astype method is to blame. **dtype** here is **int64**
> The above code does not work with 1.0.0, but used to work with at least 0.25.0.

looks like #29139 (i.e. 1.0.0)

225cc9284d1abfbcc9d13203419516575a63cc7a is the first bad commit
commit 225cc9284d1abfbcc9d13203419516575a63cc7a
Author: jbrockmendel <jbrockmendel@gmail.com>
Date:   Thu Oct 24 05:10:07 2019 -0700

    CLN: remove Block._try_coerce_arg (#29139)

cc @jbrockmendel 
also affected by this bug
Same here. However got it using df.replace({pd.NaT: None}) instead of df.where(pd.notnull(df), None)
@TomAugspurger adding the blocker tag as several users affected by this regression.
@simonjayhawkins i suppose this is ok for 1.1 as a blocker, though generally we should not simply block on things, this would delay releases indefinitly which is a much worse problem.
still broken in 1.1.3
for all those commenting 'this is still broken' - well it's an open issue

you are welcome to propose a patch
sorry, it was marked as a blocker for 1.1 and I was not sure about its status

Commit message: TST: added test for pd.where overflow error GH31687

Patch to review:
diff --git a/pandas/tests/frame/indexing/test_where.py b/pandas/tests/frame/indexing/test_where.py
index 501822f856a63..4463587f63ead 100644
--- a/pandas/tests/frame/indexing/test_where.py
+++ b/pandas/tests/frame/indexing/test_where.py
@@ -18,7 +18,10 @@
     isna,
 )
 import pandas._testing as tm
-from pandas._testing._hypothesis import OPTIONAL_ONE_OF_ALL
+from pandas._testing._hypothesis import (
+    DATETIME_JAN_1_1900_OPTIONAL_TZ,
+    OPTIONAL_ONE_OF_ALL,
+)
 
 
 @pytest.fixture(params=["default", "float_string", "mixed_float", "mixed_int"])
@@ -1017,3 +1020,15 @@ def test_where_producing_ea_cond_for_np_dtype():
         {"a": Series([pd.NA, pd.NA, 2], dtype="Int64"), "b": [np.nan, 2, 3]}
     )
     tm.assert_frame_equal(result, expected)
+
+
+@pytest.mark.parametrize(
+    "replacement", [0.001, True, "snake", DATETIME_JAN_1_1900_OPTIONAL_TZ]
+)
+def test_where_int_overflow(replacement):
+    # GH 31687
+    df = DataFrame([[1.0, 2e19, "nine"], [np.nan, 0.1, None]])
+    result = df.where(pd.notnull(df), replacement)
+    expected = DataFrame([[1.0, 2e19, "nine"], [replacement, 0.1, replacement]])
+
+    tm.assert_frame_equal(result, expected)


Code review comments:
diff_hunk: @@ -1017,3 +1020,15 @@ def test_where_producing_ea_cond_for_np_dtype():
         {"a": Series([pd.NA, pd.NA, 2], dtype="Int64"), "b": [np.nan, 2, 3]}
     )
     tm.assert_frame_equal(result, expected)
+
+
+@pytest.mark.parametrize(
+    "replacement", [0.001, True, "snake", DATETIME_JAN_1_1900_OPTIONAL_TZ]
path: pandas/tests/frame/indexing/test_where.py
text: @user2:
` DATETIME_JAN_1_1900_OPTIONAL_TZ` is gonna be a `LazyStrategy` object here, which I guess isn't quite what you wanted, I think you'll need `@user1` to use hypothesis tests
Not sure we need that though, just some date should be fine if you want to test using a date as a replacement

Also, the original issue used `None`, shall we include that?

@author:
You're right, I didn't want a LazyStrategy object! I'll change that to a date, and also include None to align better with the original issue. I was also wondering if there was already a quick way to test all scalar types rather than having to use a long parametrized list. Or would that be overkill?

@user2:
yeah I think just keeping the original snippet from the issue as a test should be fine

diff_hunk: @@ -1017,3 +1020,15 @@ def test_where_producing_ea_cond_for_np_dtype():
         {"a": Series([pd.NA, pd.NA, 2], dtype="Int64"), "b": [np.nan, 2, 3]}
     )
     tm.assert_frame_equal(result, expected)
+
+
+@pytest.mark.parametrize(
+    "replacement", [0.001, True, "snake", DATETIME_JAN_1_1900_OPTIONAL_TZ]
+)
+def test_where_int_overflow(replacement):
+    # GH 31687
+    df = DataFrame([[1.0, 2e19, "nine"], [np.nan, 0.1, None]])
path: pandas/tests/frame/indexing/test_where.py
text: @user1:
ideally for regression tests it's better to keep them as close as possible to the original - any reason to use `2e19` instead of `2e25`?

@author:
I think I reduced it down to the smallest order of magnitude that still had the problem in pandas 1.4.1, but I don't have strong feelings about using that instead of the 2e25 from the original.