Repository: pandas-dev/pandas
Pull request: API/REGR: Convert to float for index union
This restores the 0.24.x behavior of Index.union(other) between
Float and (U)Int indexes. These are again floating dtype.

left  | right | output of left.union(right)
----- | ----- | ------
int   |float  | float64
int   |uint   | object
float | uint  | float64

https://github.com/pandas-dev/pandas/issues/26778#issuecomment-505108465

Closes https://github.com/pandas-dev/pandas/issues/26778


Problem statement:
REGR: Index.union preserve numerical index for range/float and int/float combinations
Recent change in master:

```
In [11]: index = pd.RangeIndex(0, 10)

In [12]: index.union(pd.Index([0.5, 1.5]))
Out[12]: Index([0.0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0], dtype='object')
```

the dtype should be float.

cc @topper-123

Hints from issue discussion:
Yeah, I'll look into it tonight
this follows our general rule which is that dissimilar cast to object 

are you proposing a rule that we cast to float if float and numbering ?
> this follows our general rule which is that dissimilar cast to object

That is not the "general" rule, *in case of numerical* indices. Eg (released pandas):

```
In [1]: pd.Index([1, 2, 3]).union(pd.Index([.5, 1.5]))
Out[1]: Float64Index([0.5, 1.0, 1.5, 2.0, 3.0], dtype='float64')
```

Or at least, *was* not the general rule, as I see that the above example also changed on master. So @topper-123 it is not specific to RangeIndex (I was thinking it might have been related to some of the changes to it you have been doing recently).

> are you proposing a rule that we cast to float if float and numbering ?

For me, that seems very logical that int + float = float (for set ops), and this is also what we have been doing up to recently.
Possibly related to https://github.com/pandas-dev/pandas/pull/23538
If this has been done by me, it has not been on purpose. I agrre It is more logical and more efficient that unions of numerical indices return a numerical index rather than an object index
http://pandas-docs.github.io/pandas-docs-travis/whatsnew/v0.25.0.html#incompatible-index-type-unions

not averse to coercing to float if we have a float for one
we already coerce between range / int indices
Yes, that's the PR I was pointing at. 
Updating the issue to reflect the discussion.
I've bisected this and the cause of the change shown by @jorisvandenbossche is indeed #23538 (commit 20d0ad159ad12885bc4694bbb8c684897d84484e).

I also agree this is a blocker a s this should not return a object dtype, as that will likely cause slowness and cause memory-usage to increase.
cc ArtinSarraf would you be interested to look into this?
Int64Index.union(Float64Index) -> float64 introduces a slight disagreement with how we decided to handle union(int, uint) in that PR. I think we have a choice between

1. dtype stability of the output
2. full precision
3. speed

We can pick two.

With `union(int64, uint64) -> object` we chose 1 and 2, sacrificing speed.

With `union(int64, float64) -> float64` we're choosing 1 and 3, sacrificing precision of integers that cannot be represented as floating point values.

Is that what we want? With index labels, perhaps full precision is more important?

Commit message: cleanup

Patch to review:
diff --git a/pandas/core/indexes/numeric.py b/pandas/core/indexes/numeric.py
index b6c8ba588f9d6..afb4ece35dac0 100644
--- a/pandas/core/indexes/numeric.py
+++ b/pandas/core/indexes/numeric.py
@@ -7,9 +7,11 @@
 
 from pandas.core.dtypes.common import (
     is_bool, is_bool_dtype, is_dtype_equal, is_extension_array_dtype, is_float,
-    is_integer_dtype, is_scalar, needs_i8_conversion, pandas_dtype)
+    is_float_dtype, is_integer_dtype, is_scalar, needs_i8_conversion,
+    pandas_dtype)
 import pandas.core.dtypes.concat as _concat
-from pandas.core.dtypes.generic import ABCInt64Index, ABCRangeIndex
+from pandas.core.dtypes.generic import (
+    ABCFloat64Index, ABCInt64Index, ABCRangeIndex, ABCUInt64Index)
 from pandas.core.dtypes.missing import isna
 
 from pandas.core import algorithms
@@ -123,6 +125,21 @@ def insert(self, loc, item):
             item = self._na_value
         return super().insert(loc, item)
 
+    def _union(self, other, sort):
+        # float | [u]int -> float
+        # <T>   | <T>    -> T
+        # <T>   | <U>    -> object
+        needs_cast = (
+            (is_integer_dtype(self.dtype) and is_float_dtype(other.dtype)) or
+            (is_integer_dtype(other.dtype) and is_float_dtype(self.dtype))
+        )
+        if needs_cast:
+            first = self.astype("float")
+            second = other.astype("float")
+            return first._union(second, sort)
+        else:
+            return super()._union(other, sort)
+
 
 _num_index_shared_docs['class_descr'] = """
     Immutable ndarray implementing an ordered, sliceable set. The basic object
@@ -225,10 +242,24 @@ def _assert_safe_casting(cls, data, subarr):
     def _is_compatible_with_other(self, other):
         return (
             super()._is_compatible_with_other(other)
-            or all(isinstance(type(obj), (ABCInt64Index, ABCRangeIndex))
+            or all(isinstance(type(obj), (ABCInt64Index,
+                                          ABCFloat64Index,
+                                          ABCRangeIndex))
                    for obj in [self, other])
         )
 
+    def _union(self, other, sort):
+        needs_cast = (
+            (is_integer_dtype(self.dtype) and is_float_dtype(other.dtype)) or
+            (is_integer_dtype(other.dtype) and is_float_dtype(self.dtype))
+        )
+        if needs_cast:
+            first = self.astype("float")
+            second = other.astype("float")
+            return first._union(second, sort)
+        else:
+            return super()._union(other, sort)
+
 
 Int64Index._add_numeric_methods()
 Int64Index._add_logical_methods()
@@ -301,6 +332,26 @@ def _assert_safe_casting(cls, data, subarr):
                 raise TypeError('Unsafe NumPy casting, you must '
                                 'explicitly cast')
 
+    def _is_compatible_with_other(self, other):
+        return (
+            super()._is_compatible_with_other(other)
+            or all(isinstance(type(obj), (ABCUInt64Index,
+                                          ABCFloat64Index))
+                   for obj in [self, other])
+        )
+
+    def _union(self, other, sort):
+        needs_cast = (
+            (is_integer_dtype(self.dtype) and is_float_dtype(other.dtype)) or
+            (is_integer_dtype(other.dtype) and is_float_dtype(self.dtype))
+        )
+        if needs_cast:
+            first = self.astype("float")
+            second = other.astype("float")
+            return first._union(second, sort)
+        else:
+            return super()._union(other, sort)
+
 
 UInt64Index._add_numeric_methods()
 UInt64Index._add_logical_methods()
@@ -447,6 +498,16 @@ def isin(self, values, level=None):
             self._validate_index_level(level)
         return algorithms.isin(np.array(self), values)
 
+    def _is_compatible_with_other(self, other):
+        return (
+            super()._is_compatible_with_other(other)
+            or all(isinstance(type(obj), (ABCInt64Index,
+                                          ABCFloat64Index,
+                                          ABCUInt64Index,
+                                          ABCRangeIndex))
+                   for obj in [self, other])
+        )
+
 
 Float64Index._add_numeric_methods()
 Float64Index._add_logical_methods_disabled()
diff --git a/pandas/tests/indexes/test_numeric.py b/pandas/tests/indexes/test_numeric.py
index c61e0fa6d6021..ad41da29e9707 100644
--- a/pandas/tests/indexes/test_numeric.py
+++ b/pandas/tests/indexes/test_numeric.py
@@ -1118,3 +1118,27 @@ def test_join_outer(self):
         tm.assert_index_equal(res, eres)
         tm.assert_numpy_array_equal(lidx, elidx)
         tm.assert_numpy_array_equal(ridx, eridx)
+
+
+@pytest.mark.parametrize("dtype", ['int64', 'uint64'])
+def test_int_float_union_dtype(dtype):
+    # [u]int | float -> float
+    index = pd.Index([0, 2, 3], dtype=dtype)
+    other = pd.Float64Index([0.5, 1.5])
+    expected = pd.Float64Index([0.0, 0.5, 1.5, 2.0, 3.0])
+    # result = index.union(other)
+    # tm.assert_index_equal(result, expected)
+
+    result = other.union(index)
+    tm.assert_index_equal(result, expected)
+
+
+def test_range_float_union_dtype():
+    index = pd.RangeIndex(start=0, stop=3)
+    other = pd.Float64Index([0.5, 1.5])
+    result = index.union(other)
+    expected = pd.Float64Index([0.0, 0.5, 1, 1.5, 2.0])
+    tm.assert_index_equal(result, expected)
+
+    result = other.union(index)
+    tm.assert_index_equal(result, expected)
diff --git a/pandas/tests/indexes/test_setops.py b/pandas/tests/indexes/test_setops.py
index b626ced2ccb1b..e080a06e8acc2 100644
--- a/pandas/tests/indexes/test_setops.py
+++ b/pandas/tests/indexes/test_setops.py
@@ -10,12 +10,16 @@
 from pandas.core.dtypes.common import is_dtype_equal
 
 import pandas as pd
-from pandas import Int64Index, RangeIndex
+from pandas import Float64Index, Int64Index, RangeIndex, UInt64Index
+from pandas.api.types import pandas_dtype
 from pandas.tests.indexes.conftest import indices_list
 import pandas.util.testing as tm
 
 COMPATIBLE_INCONSISTENT_PAIRS = {
-    (Int64Index, RangeIndex): (tm.makeIntIndex, tm.makeRangeIndex)
+    (Int64Index, RangeIndex): (tm.makeIntIndex, tm.makeRangeIndex),
+    (Float64Index, Int64Index): (tm.makeFloatIndex, tm.makeIntIndex),
+    (Float64Index, RangeIndex): (tm.makeFloatIndex, tm.makeIntIndex),
+    (Float64Index, UInt64Index): (tm.makeFloatIndex, tm.makeUIntIndex),
 }
 
 
@@ -74,3 +78,29 @@ def test_compatible_inconsistent_pairs(idx_fact1, idx_fact2):
 
     assert res1.dtype in (idx1.dtype, idx2.dtype)
     assert res2.dtype in (idx1.dtype, idx2.dtype)
+
+
+@pytest.mark.parametrize('left, right, expected', [
+    ('int64', 'int64', 'int64'),
+    ('int64', 'uint64', 'object'),
+    ('int64', 'float64', 'float64'),
+    ('uint64', 'float64', 'float64'),
+    ('uint64', 'uint64', 'uint64'),
+    ('float64', 'float64', 'float64'),
+    ('datetime64[ns]', 'int64', 'object'),
+    ('datetime64[ns]', 'uint64', 'object'),
+    ('datetime64[ns]', 'float64', 'object'),
+    ('datetime64[ns, CET]', 'int64', 'object'),
+    ('datetime64[ns, CET]', 'uint64', 'object'),
+    ('datetime64[ns, CET]', 'float64', 'object'),
+    ('Period[D]', 'int64', 'object'),
+    ('Period[D]', 'uint64', 'object'),
+    ('Period[D]', 'float64', 'object'),
+])
+def test_union_dtypes(left, right, expected):
+    left = pandas_dtype(left)
+    right = pandas_dtype(right)
+    a = pd.Index([], dtype=left)
+    b = pd.Index([], dtype=right)
+    result = (a | b).dtype
+    assert result == expected


Code review comments:
diff_hunk: @@ -301,6 +332,26 @@ def _assert_safe_casting(cls, data, subarr):
                 raise TypeError('Unsafe NumPy casting, you must '
                                 'explicitly cast')
 
+    def _is_compatible_with_other(self, other):
+        return (
+            super()._is_compatible_with_other(other)
+            or all(isinstance(type(obj), (ABCUInt64Index,
+                                          ABCFloat64Index))
+                   for obj in [self, other])
+        )
+
+    def _union(self, other, sort):
+        needs_cast = (
+            (is_integer_dtype(self.dtype) and is_float_dtype(other.dtype)) or
+            (is_integer_dtype(other.dtype) and is_float_dtype(self.dtype))
+        )
+        if needs_cast:
path: pandas/core/indexes/numeric.py
text: @user1:
maybe makes sense to put this in a private method (the needs_cast logic) as its repeated 3 X 

@author:
Whoops, these can all use NumericIndex._union. Forgot to remove from here.