Repository: pandas-dev/pandas
Pull request: BUG: np.inf now causes Index to upcast from int to float
 - [x] closes #16957
 - [x] tests added / passed
 - [x] passes ``git diff upstream/master -u -- "*.py" | flake8 --diff``
 - [x] whatsnew entry

Previously the following code would cause an OverflowError

```python
df = pd.DataFrame(columns=[0])
df.loc[0] = 0
df.loc[np.inf] = 1
```

This patch catches the exception and allows the `pd.Int64Index` / `pd.UInt64Index` to upcast itself into a `pd.Float64Index`. 

Additionally, when I was writing tests for this patch I noticed 

```python 
np.inf in pd.Int64Index([1, 2, 3])
```

would also cause an OverflowError. I also patched this as well, and now it correctly returns False. 

------- 

I had trouble running tests on my local machine, so I'll wait for the travis checks. 

I'm also unsure the whatsnew entry is in the correct format. Let me know if that needs to be changed. 

Problem statement:
Creating a new row with np.inf as an index causes an OverflowError 
#### Code Sample, a copy-pastable example if possible

```python
# make a dataframe with one column
df = pd.DataFrame(columns=[0])
# pandas is greate because we cna use arbitrary objects to insert new
# rows and columns dynamically.
df.loc[11] = 'a'
df.loc['y', 0] = 'b'
df.loc[object(), 0] = 'c'

# BUT np.inf is not one of these object
df.loc[np.inf, 0] = 'OverflowError'
# This raises: OverflowError: cannot convert float infinity to integer


# HOWEVER, that doesn't stop us from using np.inf statically.
obj = object()
index = pd.Index([11, 'y', obj, np.inf])
df = pd.DataFrame([['a'], ['b'], ['c'], ['d']], index=index)
df.loc[np.inf, 0]

# This seem to work when we use it to look look up specific entry
df.loc[np.inf, 0]

# BUT it breaks when trying to lookup a whole row
df.loc[np.inf]

# However, this is doable with other indices
df.loc['y']
df.loc[obj]

```
#### Problem description

I came across this bug when using different threshold values to index my DataFrame. 
When I tried to dynamically insert a np.inf value, it failed with `OverflowError: cannot convert float infinity to integer`. 

At first I thought that np.inf might just not be a valid hashable object, but `{np.inf: 2}` is valid code. Furthermore, I found that I could create a `pd.Index` object with `np.inf` as a value and then set that to be the index of my DataFrame. When I did that most behavior worked as expected as long as both the row and column were specified in `df.loc[r, c]`. Calling it with just a row fails when the row is `np.inf`.

#### Expected Output

I think it is perfectly sensible that `np.inf` should be allowed as a valid index. 

I think the trouble that is happening is when you try to insert `np.inf` into an integer or object index it attempts to turn it into an integer. What probably should happen is the index should be converted into a float or object index. 

I just tested this idea:
```python
>>> df = pd.DataFrame(columns=[0])
>>> df.index
>>> print('df.index = {!r}'.format(df.index))
df.index = Index([], dtype='object')
>>> df.loc[3.3, 0] = 3
>>> df.index
df.index = Float64Index([3.3], dtype='float64')
>>> print('df.index = {!r}'.format(df.index))
>>> df.loc[np.inf, 0] = 3
>>> print(df)
        0
3.3000  3
inf     3
>>> df.loc[np.inf]
0    3
Name: inf, dtype: object
```

It seems that if you ensure the index is a float index, then `np.inf` becomes a valid key. 

So maybe the fix is just doing whatever is done when a float is given to an integer index.


#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit: None
python: 3.5.2.final.0
python-bits: 64
OS: Linux
OS-release: 4.4.0-59-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.20.3
pytest: 3.0.3
pip: 9.0.1
setuptools: 34.1.1
Cython: 0.25.2
numpy: 1.13.1
scipy: 0.19.0
xarray: None
IPython: 5.2.2
sphinx: None
patsy: 0.4.1
dateutil: 2.6.1
pytz: 2017.2
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: 2.0.2
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: 4.5.3
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.9.5
s3fs: None
pandas_gbq: None
pandas_datareader: None

</details>

Hints from issue discussion:
@Erotemic : Seems reasonable to me.  If you think you can patch it as well, a PR is welcome!

Commit message: FIX: np.inf now upcasts int Index to float

Patch to review:
diff --git a/doc/source/whatsnew/v0.21.0.txt b/doc/source/whatsnew/v0.21.0.txt
index 935e9d740b91c..4ff973046f932 100644
--- a/doc/source/whatsnew/v0.21.0.txt
+++ b/doc/source/whatsnew/v0.21.0.txt
@@ -145,6 +145,7 @@ Bug Fixes
 ~~~~~~~~~
 
 - Fixes regression in 0.20, :func:`Series.aggregate` and :func:`DataFrame.aggregate` allow dictionaries as return values again (:issue:`16741`)
+- Fixes inf upcast from integer indices in 0.20, previously :func:`Int64Index.__contains__` and `DataFrame.loc.__setitem__` raised an `OverflowError` when given `np.inf` (:issue:`16957`)
 
 Conversion
 ^^^^^^^^^^
diff --git a/pandas/core/indexes/base.py b/pandas/core/indexes/base.py
index bbbc19b36964d..5d50f961927c7 100644
--- a/pandas/core/indexes/base.py
+++ b/pandas/core/indexes/base.py
@@ -666,7 +666,7 @@ def _try_convert_to_int_index(cls, data, copy, name):
             res = data.astype('u8', copy=False)
             if (res == data).all():
                 return UInt64Index(res, copy=copy, name=name)
-        except (TypeError, ValueError):
+        except (OverflowError, TypeError, ValueError):
             pass
 
         raise ValueError
@@ -1640,7 +1640,7 @@ def __contains__(self, key):
         hash(key)
         try:
             return key in self._engine
-        except (TypeError, ValueError):
+        except (OverflowError, TypeError, ValueError):
             return False
 
     _index_shared_docs['contains'] = """
@@ -3365,7 +3365,7 @@ def _maybe_cast_indexer(self, key):
                 ckey = int(key)
                 if ckey == key:
                     key = ckey
-            except (ValueError, TypeError):
+            except (OverflowError, ValueError, TypeError):
                 pass
         return key
 
diff --git a/pandas/tests/indexing/test_indexing.py b/pandas/tests/indexing/test_indexing.py
index 9fa677eb624ae..33d0247a64b62 100644
--- a/pandas/tests/indexing/test_indexing.py
+++ b/pandas/tests/indexing/test_indexing.py
@@ -63,6 +63,35 @@ def f():
 
         pytest.raises(ValueError, f)
 
+    def test_inf_upcast(self):
+        # GH 16957
+        # We should be able to use np.inf as a key
+        # np.inf should cause an index to convert to float
+
+        # Test with np.inf in rows
+        df = pd.DataFrame(columns=[0])
+        df.loc[1] = 1
+        df.loc[2] = 2
+        df.loc[np.inf] = 3
+
+        # make sure we can look up the value
+        result = df.loc[np.inf, 0]
+        tm.assert_almost_equal(result, 3)
+
+        result = df.index
+        expected = pd.Float64Index([1, 2, np.inf])
+        tm.assert_index_equal(result, expected)
+
+        # Test with np.inf in columns
+        df = pd.DataFrame()
+        df.loc[0, 0] = 1
+        df.loc[1, 1] = 2
+        df.loc[0, np.inf] = 3
+
+        result = df.columns
+        expected = pd.Float64Index([0, 1, np.inf])
+        tm.assert_index_equal(result, expected)
+
     def test_setitem_dtype_upcast(self):
 
         # GH3216
@@ -542,6 +571,18 @@ def test_astype_assignment_with_dups(self):
         # result = df.get_dtype_counts().sort_index()
         # expected = Series({'float64': 2, 'object': 1}).sort_index()
 
+    def test_coercion_with_contains(self):
+        # Related to GH 16957
+        # Checking if Int64Index contains np.inf should catch the OverflowError
+        for val in [np.inf, np.nan]:
+            index = pd.Int64Index([1, 2, 3])
+            result = val in index
+            tm.assert_almost_equal(result, False)
+
+            index = pd.UInt64Index([1, 2, 3])
+            result = np.inf in index
+            tm.assert_almost_equal(result, False)
+
     def test_index_type_coercion(self):
 
         with catch_warnings(record=True):


Code review comments:
diff_hunk: @@ -542,6 +571,18 @@ def test_astype_assignment_with_dups(self):
         # result = df.get_dtype_counts().sort_index()
         # expected = Series({'float64': 2, 'object': 1}).sort_index()
 
+    def test_coercion_with_contains(self):
+        # Related to GH 16957
+        # Checking if Int64Index contains np.inf should catch the OverflowError
+        for val in [np.inf, np.nan]:
+            index = pd.Int64Index([1, 2, 3])
+            result = val in index
+            tm.assert_almost_equal(result, False)
path: pandas/tests/indexing/test_indexing.py
text: @user1:
I think you should be able to write:

~~~
assert val not in index
~~~

instead of the two lines above.

diff_hunk: @@ -542,6 +571,18 @@ def test_astype_assignment_with_dups(self):
         # result = df.get_dtype_counts().sort_index()
         # expected = Series({'float64': 2, 'object': 1}).sort_index()
 
+    def test_coercion_with_contains(self):
+        # Related to GH 16957
+        # Checking if Int64Index contains np.inf should catch the OverflowError
+        for val in [np.inf, np.nan]:
path: pandas/tests/indexing/test_indexing.py
text: @user1:
We can avoid this `for`-loop using `pytest` parametrization, which you can see <a href="http://doc.pytest.org/en/latest/parametrize.html">here</a>