Repository: pandas-dev/pandas
Pull request: ENH - Index set operation modifications to address issue #23525
This is a first pass at addressing the associated issue. Needs plenty of discussion, feedback and testing.

- [ ] closes #23525
- [ ] tests added / passed
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Problem statement:
Why is pd.Index.union not commutative?
#### Code Sample, a copy-pastable example if possible

```python
>>> ix = pd.Index([1,2])
>>> eix = pd.Index([])
>>> pi = pd.PeriodIndex(['19910905', '19910906'], freq='D')

# Pair 1
>>> pi.union(eix)
ValueError: can only call with other PeriodIndex-ed objects
>>> eix.union(pi)
PeriodIndex(['1991-09-05', '1991-09-06'], dtype='period[D]', freq='D')

# Pair 2
>>> pi.union(ix)
ValueError: can only call with other PeriodIndex-ed objects
>>> ix.union(pi)
Index([1, 2, 1991-09-05, 1991-09-06], dtype='object')
```
#### Problem description
Conceptually I would imagine a union operation to be commutative. I was just wondering if there was an deliberate rationale behind not implementing pd.Index._assert_can_do_setop to only fail if the complementary self._assert_can_do_setop also fails.

This behavior also leads to some unexpected behaviors in `pd.concat`. For example:

    >>> df1 = df1 = pd.DataFrame([[1,2,3],[1,2,3]], index=pd.PeriodIndex(['19910905', '19910906'], freq='D'))
    >>> df2 = pd.DataFrame()
    >>> pd.concat([df1, df2], axis=1, keys=['a', 'b'])
    ValueError: can only call with other PeriodIndex-ed objects
    >>> pd.concat([df2, df1], axis=1, keys=['a', 'b'])
    Works!

Additionally (and perhaps this should be raised as a separate issue) should the specific implementation of `pd.PeriodIndex._assert_can_do_setop` not raise if the `other` index is empty? Since `pd.Index([]).union(<instance of pd.PeriodIndex>)` results in an instance of `pd.PeriodIndex`.

Hints from issue discussion:
cc @jreback 
we don't ignore empties. actually these should all convert to object dtype (and work).
Agreed.
@jreback - just to clarify - are you suggesting that:

1) the union of an empty index with any other index should not result in an index of the same type as the "other" index.
e.g.

```python
>>> pd.Index([]).union(pd.period_range('19910905', periods=2))
Current: PeriodIndex(['1991-09-05', '1991-09-06'], dtype='period[D]', freq='D')
Desired: Index([1991-09-05, 1991-09-06], dtype='object')
>>> pd.Index([]).union(pd.interval_range(start=0, end=5))
Current: IntervalIndex([(0, 1], (1, 2]]
                       closed='right',
                       dtype='interval[int64]')
Desired: Index([(0, 1], (1, 2]], dtype='object')
```

2) Any index should be able to form a union with any other type, as long as the resultant dtype is `object`?
e.g.

```python
>>> pd.period_range('19910905', periods=2).union(pd.Index([1,2,3]))
Current: ValueError
Desired: Index([1991-09-05, 1991-09-06, 1, 2, 3], dtype='object')
```

2b) And should this only be the case for which the `_assert_can_do_setop` is valid for at least one of the indexes?
e.g.
Consider the following pairs:
**Index/PeriodIndex** - Index can do setop with PeriodIndex, but not vice-versa.
**IntervalIndex/PeriodIndex** - neither index can do a setop with the other - should this still work and result in an object dtype?

 

yes to both
Thoughts on 2b?
no

like indexes are combined to form the same type
unlike are coerced to object and then combine

it’s likely that the coercion logic is not fully handling things

we should not have any special cases - these are general ops
@jreback - how should the union of Int64Index and RangeIndex behave ideally? Should they also result in an object dtype or behave the same as they do now?
RangeIndex should always be just an optimization of Int64Index. So you
would return a RangeIndex if possible, else an Int64Index.

On Thu, Nov 8, 2018 at 9:03 PM ArtinSarraf <notifications@github.com> wrote:

> @jreback <https://github.com/jreback> - how should the union of
> Int64Index and RangeIndex behave ideally? Should they also result in an
> object dtype or behave the same as they do now?
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/issues/23525#issuecomment-437234514>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIu9_YxMmMlL0xNVPSc523ZESwKX1ks5utPCKgaJpZM4YPp62>
> .
>

Commit message: BUG - remove print

Patch to review:
diff --git a/doc/source/whatsnew/v0.24.0.rst b/doc/source/whatsnew/v0.24.0.rst
index a49ea2cf493a6..98909471bbe50 100644
--- a/doc/source/whatsnew/v0.24.0.rst
+++ b/doc/source/whatsnew/v0.24.0.rst
@@ -859,6 +859,37 @@ Bug in :func:`DataFrame.to_dict` raises ``ValueError`` when used with
 
     df.to_dict(orient='index')
 
+.. _whatsnew_0240.api_breaking.incompatible_index_unions
+
+Incompatible Index Type Unions
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+When performing an :func:`Index.union` operation, between objects of incompatible dtypes,
+the result will be a base :class:`Index` of dtype `object`. This behavior holds true for
+unions between :class:`Index` objects that previously would have been prohibited. The dtype
+of empty :class:`Index` objects will now be evaluated before performing union operations
+rather than simply returning the other :class:`Index` object. :func:`Index.union` can now be
+considered commutative, such that ``A.union(B) == B.union(A)`` (:issue:`23525`).
+
+
+*Previous Behavior*:
+
+.. code-block:: ipython
+
+    In [1]: pd.period_range('19910905', periods=2).union(pd.Int64Index([1, 2, 3]))
+    ...
+    ValueError: can only call with other PeriodIndex-ed objects
+
+    In [2]: pd.Index([]).astype(str).union(pd.Index([1, 2, 3]))
+    Out[2]: Int64Index([1, 2, 3], dtype='int64')
+
+*Current Behavior*:
+
+.. ipython:: python
+
+    pd.period_range('19910905', periods=2).union(pd.Int64Index([1, 2, 3]))
+    pd.Index([]).astype(str).union(pd.Index([1, 2, 3]))
+
 .. _whatsnew_0240.api.datetimelike.normalize:
 
 Tick DateOffset Normalize Restrictions
diff --git a/pandas/core/indexes/base.py b/pandas/core/indexes/base.py
index cf813f4c3030b..6846e75258f21 100644
--- a/pandas/core/indexes/base.py
+++ b/pandas/core/indexes/base.py
@@ -19,11 +19,10 @@
     ensure_categorical, ensure_int64, ensure_object, ensure_platform_int,
     is_bool, is_bool_dtype, is_categorical, is_categorical_dtype,
     is_datetime64_any_dtype, is_datetime64tz_dtype, is_dtype_equal,
-    is_dtype_union_equal, is_extension_array_dtype, is_float, is_float_dtype,
-    is_hashable, is_integer, is_integer_dtype, is_interval_dtype, is_iterator,
-    is_list_like, is_object_dtype, is_period_dtype, is_scalar,
-    is_signed_integer_dtype, is_timedelta64_dtype, is_unsigned_integer_dtype,
-    pandas_dtype)
+    is_extension_array_dtype, is_float, is_float_dtype, is_hashable,
+    is_integer, is_integer_dtype, is_interval_dtype, is_iterator, is_list_like,
+    is_object_dtype, is_period_dtype, is_scalar, is_signed_integer_dtype,
+    is_timedelta64_dtype, is_unsigned_integer_dtype, pandas_dtype)
 import pandas.core.dtypes.concat as _concat
 from pandas.core.dtypes.generic import (
     ABCDataFrame, ABCDateOffset, ABCDatetimeArray, ABCIndexClass,
@@ -2245,6 +2244,25 @@ def _get_reconciled_name_object(self, other):
             return self._shallow_copy(name=name)
         return self
 
+    def _union_incompatible_dtypes(self, other, sort):
+        """
+        Casts this and other index to object dtype to allow the formation
+        of a union between incompatible types.
+        """
+        this = self.astype(object, copy=False)
+        # cast to Index for when `other` is list-like
+        other = Index(other).astype(object, copy=False)
+        return Index.union(this, other, sort=sort).astype(object, copy=False)
+
+    def _is_compatible_with_other(self, other):
+        """
+        Check whether this and the other dtype are compatible with each other.
+        Meaning a union can be formed between them without needing to be cast
+        to dtype object.
+        """
+        return (type(self) is type(other)
+                and is_dtype_equal(self.dtype, other.dtype))
+
     def _validate_sort_keyword(self, sort):
         if sort not in [None, False]:
             raise ValueError("The 'sort' keyword only takes the values of "
@@ -2254,6 +2272,9 @@ def union(self, other, sort=None):
         """
         Form the union of two Index objects.
 
+        If the Index objects are incompatible, both Index objects will be
+        cast to dtype('object') first.
+
         Parameters
         ----------
         other : Index or array-like
@@ -2287,26 +2308,32 @@ def union(self, other, sort=None):
         >>> idx2 = pd.Index([3, 4, 5, 6])
         >>> idx1.union(idx2)
         Int64Index([1, 2, 3, 4, 5, 6], dtype='int64')
+
+        >>> idx1 = pd.Index(['a', 'b', 'c', 'd'])
+        >>> idx2 = pd.Index([1, 2, 3, 4])
+        >>> idx1.union(idx2)
+        Index(['a', 'b', 'c', 'd', 1, 2, 3, 4], dtype='object')
         """
         self._validate_sort_keyword(sort)
         self._assert_can_do_setop(other)
-        other = ensure_index(other)
 
-        if len(other) == 0 or self.equals(other):
+        if not self._is_compatible_with_other(other):
+            return self._union_incompatible_dtypes(other, sort=sort)
+
+        return self._union(other, sort=sort)
+
+    def _union(self, other, sort):
+        """
+        Specific union logic should go here. In subclasses union behavior
+        should be overwritten here rather than in `self.union`
+        """
+
+        if not len(other) or self.equals(other):
             return self._get_reconciled_name_object(other)
 
-        if len(self) == 0:
+        if not len(self):
             return other._get_reconciled_name_object(self)
 
-        # TODO: is_dtype_union_equal is a hack around
-        # 1. buggy set ops with duplicates (GH #13432)
-        # 2. CategoricalIndex lacking setops (GH #10186)
-        # Once those are fixed, this workaround can be removed
-        if not is_dtype_union_equal(self.dtype, other.dtype):
-            this = self.astype('O')
-            other = other.astype('O')
-            return this.union(other, sort=sort)
-
         # TODO(EA): setops-refactor, clean all this up
         if is_period_dtype(self) or is_datetime64tz_dtype(self):
             lvals = self._ndarray_values
@@ -2353,6 +2380,7 @@ def union(self, other, sort=None):
     def _wrap_setop_result(self, other, result):
         return self._constructor(result, name=get_op_result_name(self, other))
 
+    # TODO: standardize return type of non-union setops type(self vs other)
     def intersection(self, other, sort=False):
         """
         Form the intersection of two Index objects.
diff --git a/pandas/core/indexes/datetimes.py b/pandas/core/indexes/datetimes.py
index df91c71cfe238..6dff44cdeb495 100644
--- a/pandas/core/indexes/datetimes.py
+++ b/pandas/core/indexes/datetimes.py
@@ -460,24 +460,9 @@ def _formatter_func(self):
     # --------------------------------------------------------------------
     # Set Operation Methods
 
-    def union(self, other):
-        """
-        Specialized union for DatetimeIndex objects. If combine
-        overlapping ranges with the same DateOffset, will be much
-        faster than Index.union
-
-        Parameters
-        ----------
-        other : DatetimeIndex or array-like
-
-        Returns
-        -------
-        y : Index or DatetimeIndex
-        """
-        self._assert_can_do_setop(other)
-
-        if len(other) == 0 or self.equals(other) or len(self) == 0:
-            return super(DatetimeIndex, self).union(other)
+    def _union(self, other, sort):
+        if not len(other) or self.equals(other) or not len(self):
+            return super(DatetimeIndex, self)._union(other, sort=sort)
 
         if not isinstance(other, DatetimeIndex):
             try:
@@ -490,7 +475,7 @@ def union(self, other):
         if this._can_fast_union(other):
             return this._fast_union(other)
         else:
-            result = Index.union(this, other)
+            result = Index._union(this, other, sort=sort)
             if isinstance(result, DatetimeIndex):
                 # TODO: we shouldn't be setting attributes like this;
                 #  in all the tests this equality already holds
diff --git a/pandas/core/indexes/interval.py b/pandas/core/indexes/interval.py
index 2c63fe33c57fe..3d2cc3ea690c6 100644
--- a/pandas/core/indexes/interval.py
+++ b/pandas/core/indexes/interval.py
@@ -967,19 +967,6 @@ def insert(self, loc, item):
         new_right = self.right.insert(loc, right_insert)
         return self._shallow_copy(new_left, new_right)
 
-    def _as_like_interval_index(self, other):
-        self._assert_can_do_setop(other)
-        other = ensure_index(other)
-        if not isinstance(other, IntervalIndex):
-            msg = ('the other index needs to be an IntervalIndex too, but '
-                   'was type {}').format(other.__class__.__name__)
-            raise TypeError(msg)
-        elif self.closed != other.closed:
-            msg = ('can only do set operations between two IntervalIndex '
-                   'objects that are closed on the same side')
-            raise ValueError(msg)
-        return other
-
     def _concat_same_dtype(self, to_concat, name):
         """
         assert that we all have the same .closed
@@ -1095,7 +1082,17 @@ def overlaps(self, other):
 
     def _setop(op_name, sort=None):
         def func(self, other, sort=sort):
-            other = self._as_like_interval_index(other)
+            self._assert_can_do_setop(other)
+            other = ensure_index(other)
+            if not isinstance(other, IntervalIndex):
+                result = getattr(self.astype(object), op_name)(other)
+                if op_name in ('difference',):
+                    result = result.astype(self.dtype)
+                return result
+            elif self.closed != other.closed:
+                msg = ('can only do set operations between two IntervalIndex '
+                       'objects that are closed on the same side')
+                raise ValueError(msg)
 
             # GH 19016: ensure set op will not return a prohibited dtype
             subtypes = [self.dtype.subtype, other.dtype.subtype]
@@ -1117,6 +1114,7 @@ def func(self, other, sort=sort):
 
             return type(self).from_tuples(result, closed=self.closed,
                                           name=result_name)
+
         return func
 
     @property
diff --git a/pandas/core/indexes/numeric.py b/pandas/core/indexes/numeric.py
index 379464f4fced6..5633bae87cabe 100644
--- a/pandas/core/indexes/numeric.py
+++ b/pandas/core/indexes/numeric.py
@@ -10,6 +10,7 @@
     is_bool, is_bool_dtype, is_dtype_equal, is_extension_array_dtype, is_float,
     is_integer_dtype, is_scalar, needs_i8_conversion, pandas_dtype)
 import pandas.core.dtypes.concat as _concat
+from pandas.core.dtypes.generic import ABCInt64Index, ABCRangeIndex
 from pandas.core.dtypes.missing import isna
 
 from pandas.core import algorithms
@@ -224,6 +225,13 @@ def _assert_safe_casting(cls, data, subarr):
                 raise TypeError('Unsafe NumPy casting, you must '
                                 'explicitly cast')
 
+    def _is_compatible_with_other(self, other):
+        return (
+            super(Int64Index, self)._is_compatible_with_other(other)
+            or all(isinstance(type(obj), (ABCInt64Index, ABCRangeIndex))
+                   for obj in [self, other])
+        )
+
 
 Int64Index._add_numeric_methods()
 Int64Index._add_logical_methods()
diff --git a/pandas/core/indexes/period.py b/pandas/core/indexes/period.py
index a4bd7f9017eb4..193c11a8d08f0 100644
--- a/pandas/core/indexes/period.py
+++ b/pandas/core/indexes/period.py
@@ -793,6 +793,11 @@ def join(self, other, how='left', level=None, return_indexers=False,
         """
         self._assert_can_do_setop(other)
 
+        if not isinstance(other, PeriodIndex):
+            return self.astype(object).join(other, how=how, level=level,
+                                            return_indexers=return_indexers,
+                                            sort=sort)
+
         result = Int64Index.join(self, other, how=how, level=level,
                                  return_indexers=return_indexers,
                                  sort=sort)
@@ -805,10 +810,9 @@ def join(self, other, how='left', level=None, return_indexers=False,
     def _assert_can_do_setop(self, other):
         super(PeriodIndex, self)._assert_can_do_setop(other)
 
-        if not isinstance(other, PeriodIndex):
-            raise ValueError('can only call with other PeriodIndex-ed objects')
-
-        if self.freq != other.freq:
+        # *Can't* use PeriodIndexes of different freqs
+        # *Can* use PeriodIndex/DatetimeIndex
+        if isinstance(other, PeriodIndex) and self.freq != other.freq:
             msg = DIFFERENT_FREQ.format(cls=type(self).__name__,
                                         own_freq=self.freqstr,
                                         other_freq=other.freqstr)
diff --git a/pandas/core/indexes/range.py b/pandas/core/indexes/range.py
index 5aafe9734b6a0..8f6fbfb91a56c 100644
--- a/pandas/core/indexes/range.py
+++ b/pandas/core/indexes/range.py
@@ -438,21 +438,9 @@ def _extended_gcd(self, a, b):
             old_t, t = t, old_t - quotient * t
         return old_r, old_s, old_t
 
-    def union(self, other):
-        """
-        Form the union of two Index objects and sorts if possible
-
-        Parameters
-        ----------
-        other : Index or array-like
-
-        Returns
-        -------
-        union : Index
-        """
-        self._assert_can_do_setop(other)
-        if len(other) == 0 or self.equals(other) or len(self) == 0:
-            return super(RangeIndex, self).union(other)
+    def _union(self, other, sort):
+        if not len(other) or self.equals(other) or not len(self):
+            return super(RangeIndex, self)._union(other, sort=sort)
 
         if isinstance(other, RangeIndex):
             start_s, step_s = self._start, self._step
@@ -491,7 +479,7 @@ def union(self, other):
                         (end_s - step_o <= end_o)):
                     return RangeIndex(start_r, end_r + step_o, step_o)
 
-        return self._int64index.union(other)
+        return self._int64index._union(other, sort=sort)
 
     @Appender(_index_shared_docs['join'])
     def join(self, other, how='left', level=None, return_indexers=False,
diff --git a/pandas/core/indexes/timedeltas.py b/pandas/core/indexes/timedeltas.py
index 830925535dab1..ba684a0516f03 100644
--- a/pandas/core/indexes/timedeltas.py
+++ b/pandas/core/indexes/timedeltas.py
@@ -333,24 +333,9 @@ def astype(self, dtype, copy=True):
             return Index(result.astype('i8'), name=self.name)
         return DatetimeIndexOpsMixin.astype(self, dtype, copy=copy)
 
-    def union(self, other):
-        """
-        Specialized union for TimedeltaIndex objects. If combine
-        overlapping ranges with the same DateOffset, will be much
-        faster than Index.union
-
-        Parameters
-        ----------
-        other : TimedeltaIndex or array-like
-
-        Returns
-        -------
-        y : Index or TimedeltaIndex
-        """
-        self._assert_can_do_setop(other)
-
+    def _union(self, other, sort):
         if len(other) == 0 or self.equals(other) or len(self) == 0:
-            return super(TimedeltaIndex, self).union(other)
+            return super(TimedeltaIndex, self)._union(other, sort=sort)
 
         if not isinstance(other, TimedeltaIndex):
             try:
@@ -362,7 +347,7 @@ def union(self, other):
         if this._can_fast_union(other):
             return this._fast_union(other)
         else:
-            result = Index.union(this, other)
+            result = Index._union(this, other, sort=sort)
             if isinstance(result, TimedeltaIndex):
                 if result.freq is None:
                     result.freq = to_offset(result.inferred_freq)
diff --git a/pandas/tests/indexes/common.py b/pandas/tests/indexes/common.py
index 6d29c147c4a4a..8fc4cb0c4b57e 100644
--- a/pandas/tests/indexes/common.py
+++ b/pandas/tests/indexes/common.py
@@ -444,11 +444,7 @@ def test_intersection_base(self):
             cases = [klass(second.values)
                      for klass in [np.array, Series, list]]
             for case in cases:
-                if isinstance(idx, PeriodIndex):
-                    msg = "can only call with other PeriodIndex-ed objects"
-                    with pytest.raises(ValueError, match=msg):
-                        first.intersection(case)
-                elif isinstance(idx, CategoricalIndex):
+                if isinstance(idx, CategoricalIndex):
                     pass
                 else:
                     result = first.intersection(case)
@@ -471,11 +467,7 @@ def test_union_base(self):
             cases = [klass(second.values)
                      for klass in [np.array, Series, list]]
             for case in cases:
-                if isinstance(idx, PeriodIndex):
-                    msg = "can only call with other PeriodIndex-ed objects"
-                    with pytest.raises(ValueError, match=msg):
-                        first.union(case)
-                elif isinstance(idx, CategoricalIndex):
+                if isinstance(idx, CategoricalIndex):
                     pass
                 else:
                     result = first.union(case)
@@ -503,11 +495,7 @@ def test_difference_base(self, sort):
             cases = [klass(second.values)
                      for klass in [np.array, Series, list]]
             for case in cases:
-                if isinstance(idx, PeriodIndex):
-                    msg = "can only call with other PeriodIndex-ed objects"
-                    with pytest.raises(ValueError, match=msg):
-                        first.difference(case, sort)
-                elif isinstance(idx, CategoricalIndex):
+                if isinstance(idx, CategoricalIndex):
                     pass
                 elif isinstance(idx, (DatetimeIndex, TimedeltaIndex)):
                     assert result.__class__ == answer.__class__
@@ -537,11 +525,7 @@ def test_symmetric_difference(self):
             cases = [klass(second.values)
                      for klass in [np.array, Series, list]]
             for case in cases:
-                if isinstance(idx, PeriodIndex):
-                    msg = "can only call with other PeriodIndex-ed objects"
-                    with pytest.raises(ValueError, match=msg):
-                        first.symmetric_difference(case)
-                elif isinstance(idx, CategoricalIndex):
+                if isinstance(idx, CategoricalIndex):
                     pass
                 else:
                     result = first.symmetric_difference(case)
diff --git a/pandas/tests/indexes/conftest.py b/pandas/tests/indexes/conftest.py
index e82cce873e75c..acd2497ceda75 100644
--- a/pandas/tests/indexes/conftest.py
+++ b/pandas/tests/indexes/conftest.py
@@ -7,23 +7,25 @@
 from pandas.core.indexes.api import Index, MultiIndex
 import pandas.util.testing as tm
 
-
-@pytest.fixture(params=[tm.makeUnicodeIndex(100),
-                        tm.makeStringIndex(100),
-                        tm.makeDateIndex(100),
-                        tm.makePeriodIndex(100),
-                        tm.makeTimedeltaIndex(100),
-                        tm.makeIntIndex(100),
-                        tm.makeUIntIndex(100),
-                        tm.makeRangeIndex(100),
-                        tm.makeFloatIndex(100),
-                        Index([True, False]),
-                        tm.makeCategoricalIndex(100),
-                        Index([]),
-                        MultiIndex.from_tuples(lzip(
-                            ['foo', 'bar', 'baz'], [1, 2, 3])),
-                        Index([0, 0, 1, 1, 2, 2])],
-                ids=lambda x: type(x).__name__)
+indices_list = [tm.makeUnicodeIndex(100),
+                tm.makeStringIndex(100),
+                tm.makeDateIndex(100),
+                tm.makePeriodIndex(100),
+                tm.makeTimedeltaIndex(100),
+                tm.makeIntIndex(100),
+                tm.makeUIntIndex(100),
+                tm.makeRangeIndex(100),
+                tm.makeFloatIndex(100),
+                Index([True, False]),
+                tm.makeCategoricalIndex(100),
+                tm.makeIntervalIndex(100),
+                Index([]),
+                MultiIndex.from_tuples(lzip(
+                    ['foo', 'bar', 'baz'], [1, 2, 3])),
+                Index([0, 0, 1, 1, 2, 2])]
+
+
+@pytest.fixture(params=indices_list, ids=lambda x: type(x).__name__)
 def indices(request):
     return request.param
 
diff --git a/pandas/tests/indexes/datetimes/test_datetime.py b/pandas/tests/indexes/datetimes/test_datetime.py
index e1ba0e1708442..aa647380e9fa1 100644
--- a/pandas/tests/indexes/datetimes/test_datetime.py
+++ b/pandas/tests/indexes/datetimes/test_datetime.py
@@ -301,9 +301,9 @@ def test_join_with_period_index(self, join_type):
             c_idx_type='p', r_idx_type='dt')
         s = df.iloc[:5, 0]
 
-        msg = 'can only call with other PeriodIndex-ed objects'
-        with pytest.raises(ValueError, match=msg):
-            df.columns.join(s.index, how=join_type)
+        expected = df.columns.astype('O').join(s.index, how=join_type)
+        result = df.columns.join(s.index, how=join_type)
+        tm.assert_index_equal(expected, result)
 
     def test_factorize(self):
         idx1 = DatetimeIndex(['2014-01', '2014-01', '2014-02', '2014-02',
diff --git a/pandas/tests/indexes/datetimes/test_setops.py b/pandas/tests/indexes/datetimes/test_setops.py
index 19009e45ee83a..1d3050c13a4da 100644
--- a/pandas/tests/indexes/datetimes/test_setops.py
+++ b/pandas/tests/indexes/datetimes/test_setops.py
@@ -29,10 +29,13 @@ def test_union2(self):
         assert tm.equalContents(union, everything)
 
         # GH 10149
+        expected = first.astype('O').union(
+            pd.Index(second.values, dtype='O')
+        ).astype('O')
         cases = [klass(second.values) for klass in [np.array, Series, list]]
         for case in cases:
             result = first.union(case)
-            assert tm.equalContents(result, everything)
+            assert tm.equalContents(result, expected)
 
     @pytest.mark.parametrize("tz", tz)
     def test_union(self, tz):
@@ -267,11 +270,12 @@ def test_datetimeindex_union_join_empty(self):
         empty = Index([])
 
         result = dti.union(empty)
-        assert isinstance(result, DatetimeIndex)
-        assert result is result
+        expected = dti.astype('O')
+        tm.assert_index_equal(result, expected)
 
         result = dti.join(empty)
         assert isinstance(result, DatetimeIndex)
+        tm.assert_index_equal(result, dti)
 
     def test_join_nonunique(self):
         idx1 = to_datetime(['2012-11-06 16:00:11.477563',
diff --git a/pandas/tests/indexes/datetimes/test_timezones.py b/pandas/tests/indexes/datetimes/test_timezones.py
index 12c1b15733895..7e8d7a33056f7 100644
--- a/pandas/tests/indexes/datetimes/test_timezones.py
+++ b/pandas/tests/indexes/datetimes/test_timezones.py
@@ -1072,7 +1072,10 @@ def test_dti_union_aware(self):
                           tz="US/Eastern")
 
         result = rng.union(rng2)
-        assert result.tz.zone == 'UTC'
+        expected = rng.astype('O').union(rng2.astype('O'))
+        tm.assert_index_equal(result, expected)
+        assert result[0].tz.zone == 'US/Central'
+        assert result[-1].tz.zone == 'US/Eastern'
 
     @pytest.mark.parametrize('tz', [None, 'UTC', "US/Central",
                                     dateutil.tz.tzoffset(None, -28800)])
diff --git a/pandas/tests/indexes/interval/test_interval.py b/pandas/tests/indexes/interval/test_interval.py
index e4f25ff143273..bf6f5b6fbd5b2 100644
--- a/pandas/tests/indexes/interval/test_interval.py
+++ b/pandas/tests/indexes/interval/test_interval.py
@@ -878,15 +878,18 @@ def test_symmetric_difference(self, closed, sort):
     @pytest.mark.parametrize('op_name', [
         'union', 'intersection', 'difference', 'symmetric_difference'])
     @pytest.mark.parametrize("sort", [None, False])
-    def test_set_operation_errors(self, closed, op_name, sort):
+    def test_set_incompatible_types(self, closed, op_name, sort):
         index = self.create_index(closed=closed)
         set_op = getattr(index, op_name)
 
+        # TODO: standardize return type of non-union setops type(self vs other)
         # non-IntervalIndex
-        msg = ('the other index needs to be an IntervalIndex too, but '
-               'was type Int64Index')
-        with pytest.raises(TypeError, match=msg):
-            set_op(Index([1, 2, 3]), sort=sort)
+        if op_name == 'difference':
+            expected = index
+        else:
+            expected = getattr(index.astype('O'), op_name)(Index([1, 2, 3]))
+        result = set_op(Index([1, 2, 3]), sort=sort)
+        tm.assert_index_equal(result, expected)
 
         # mixed closed
         msg = ('can only do set operations between two IntervalIndex objects '
diff --git a/pandas/tests/indexes/period/test_setops.py b/pandas/tests/indexes/period/test_setops.py
index bf29edad4841e..068733839e003 100644
--- a/pandas/tests/indexes/period/test_setops.py
+++ b/pandas/tests/indexes/period/test_setops.py
@@ -120,10 +120,6 @@ def test_union_misc(self, sort):
         with pytest.raises(period.IncompatibleFrequency):
             index.union(index2, sort=sort)
 
-        msg = 'can only call with other PeriodIndex-ed objects'
-        with pytest.raises(ValueError, match=msg):
-            index.join(index.to_timestamp())
-
         index3 = period_range('1/1/2000', '1/20/2000', freq='2D')
         with pytest.raises(period.IncompatibleFrequency):
             index.join(index3)
diff --git a/pandas/tests/indexes/test_base.py b/pandas/tests/indexes/test_base.py
index c99007cef90d4..1a0e6c8130237 100644
--- a/pandas/tests/indexes/test_base.py
+++ b/pandas/tests/indexes/test_base.py
@@ -889,6 +889,8 @@ def test_union_identity(self, sort):
         # i.e. identity is not preserved when sort is True
         assert (union is first) is (not sort)
 
+        # This should no longer be the same object, since [] is not consistent,
+        # both objects will be recast to dtype('O')
         union = first.union([], sort=sort)
         assert (union is first) is (not sort)
 
diff --git a/pandas/tests/indexes/test_setops.py b/pandas/tests/indexes/test_setops.py
new file mode 100644
index 0000000000000..359addf33498c
--- /dev/null
+++ b/pandas/tests/indexes/test_setops.py
@@ -0,0 +1,70 @@
+'''
+The tests in this package are to ensure the proper resultant dtypes of
+set operations.
+'''
+import itertools as it
+
+import numpy as np
+import pytest
+
+from pandas.core.dtypes.common import is_dtype_equal
+
+import pandas as pd
+from pandas import Int64Index, RangeIndex
+from pandas.tests.indexes.conftest import indices_list
+import pandas.util.testing as tm
+
+COMPATIBLE_INCONSISTENT_PAIRS = {
+    (Int64Index, RangeIndex): (tm.makeIntIndex, tm.makeRangeIndex)
+}
+
+
+def test_union_same_types(indices):
+    # Union with a non-unique, non-monotonic index raises error
+    # Only needed for bool index factory
+    idx1 = indices.sort_values()
+    idx2 = indices.sort_values()
+    assert idx1.union(idx2).dtype == idx1.dtype
+
+
+@pytest.mark.parametrize(
+    'idx1,idx2',
+    list(it.combinations(indices_list, 2))
+)
+def test_union_different_types(idx1, idx2):
+    # GH 23525
+    pair = tuple(sorted([type(idx1), type(idx2)], key=lambda x: str(x)))
+    if pair in COMPATIBLE_INCONSISTENT_PAIRS:
+        return
+
+    if any(isinstance(idx, pd.MultiIndex) for idx in [idx1, idx2]):
+        return
+
+    if is_dtype_equal(idx1.dtype, idx2.dtype):
+        return
+
+    # A union with a CategoricalIndex (even as dtype('O')) and a
+    # non-CategoricalIndex can only be made if both indices are monotonic.
+    # This is true before this PR as well.
+
+    # Union with a non-unique, non-monotonic index raises error
+    # This applies to the boolean index
+    idx1 = idx1.sort_values()
+    idx2 = idx2.sort_values()
+
+    assert idx1.union(idx2).dtype == np.dtype('O')
+    assert idx2.union(idx1).dtype == np.dtype('O')
+
+
+@pytest.mark.parametrize('idx_fact1,idx_fact2',
+                         COMPATIBLE_INCONSISTENT_PAIRS.values())
+def test_compatible_inconsistent_pairs(idx_fact1, idx_fact2):
+    # GH 23525
+    idx1 = idx_fact1(10)
+    idx2 = idx_fact2(20)
+
+    res1 = idx1.union(idx2)
+    res2 = idx2.union(idx1)
+
+    assert res1.dtype in (idx1.dtype, idx2.dtype)
+    assert res2.dtype in (idx1.dtype, idx2.dtype)
diff --git a/pandas/tests/reshape/test_concat.py b/pandas/tests/reshape/test_concat.py
index a186d32ed8800..79338b47f660a 100644
--- a/pandas/tests/reshape/test_concat.py
+++ b/pandas/tests/reshape/test_concat.py
@@ -961,22 +961,22 @@ def test_append_different_columns_types_raises(
         df = pd.DataFrame([[1, 2, 3], [4, 5, 6]], columns=index_can_append)
         ser = pd.Series([7, 8, 9], index=index_cannot_append_with_other,
                         name=2)
-        msg = ("the other index needs to be an IntervalIndex too, but was"
+        msg = (r"unorderable types: (Interval|int)\(\) (<|>) "
+               r"(int|long|float|str)\(\)|"
+               r"Expected tuple, got (int|long|float|str)|"
+               r"Cannot compare type 'Timestamp' with type '(int|long)'|"
+               r"'(<|>)' not supported between instances of 'int' and 'str'|"
+               r"the other index needs to be an IntervalIndex too, but was"
                r" type {}|"
                r"object of type '(int|long|float|Timestamp)' has no len\(\)|"
                "Expected tuple, got str")
-        with pytest.raises(TypeError, match=msg.format(
-                index_can_append.__class__.__name__)):
+        with pytest.raises(TypeError, match=msg):
             df.append(ser)
 
         df = pd.DataFrame([[1, 2, 3], [4, 5, 6]],
                           columns=index_cannot_append_with_other)
         ser = pd.Series([7, 8, 9], index=index_can_append, name=2)
-        msg = (r"unorderable types: (Interval|int)\(\) > "
-               r"(int|long|float|str)\(\)|"
-               r"Expected tuple, got (int|long|float|str)|"
-               r"Cannot compare type 'Timestamp' with type '(int|long)'|"
-               r"'>' not supported between instances of 'int' and 'str'")
+
         with pytest.raises(TypeError, match=msg):
             df.append(ser)
 
@@ -2058,7 +2058,8 @@ def test_concat_empty_series(self):
         s1 = pd.Series([1, 2, 3], name='x')
         s2 = pd.Series(name='y')
         res = pd.concat([s1, s2], axis=1)
-        exp = pd.DataFrame({'x': [1, 2, 3], 'y': [np.nan, np.nan, np.nan]})
+        exp = pd.DataFrame({'x': [1, 2, 3], 'y': [np.nan, np.nan, np.nan]},
+                           index=pd.Index([0, 1, 2], dtype='O'))
         tm.assert_frame_equal(res, exp)
 
         s1 = pd.Series([1, 2, 3], name='x')
@@ -2073,7 +2074,8 @@ def test_concat_empty_series(self):
         s2 = pd.Series(name=None)
         res = pd.concat([s1, s2], axis=1)
         exp = pd.DataFrame({'x': [1, 2, 3], 0: [np.nan, np.nan, np.nan]},
-                           columns=['x', 0])
+                           columns=['x', 0],
+                           index=pd.Index([0, 1, 2], dtype='O'))
         tm.assert_frame_equal(res, exp)
 
     @pytest.mark.parametrize('tz', [None, 'UTC'])
diff --git a/pandas/tests/series/test_combine_concat.py b/pandas/tests/series/test_combine_concat.py
index 45e3dffde60f7..e4a06dd41065f 100644
--- a/pandas/tests/series/test_combine_concat.py
+++ b/pandas/tests/series/test_combine_concat.py
@@ -103,6 +103,7 @@ def test_combine_first(self):
         # corner case
         s = Series([1., 2, 3], index=[0, 1, 2])
         result = s.combine_first(Series([], index=[]))
+        s.index = s.index.astype('O')
         assert_series_equal(s, result)
 
     def test_update(self):
diff --git a/pandas/tests/series/test_missing.py b/pandas/tests/series/test_missing.py
index 985288c439917..f2d55d6d1f66d 100644
--- a/pandas/tests/series/test_missing.py
+++ b/pandas/tests/series/test_missing.py
@@ -892,7 +892,7 @@ def test_interpolate_pchip(self):
 
         # interpolate at new_index
         new_index = ser.index.union(Index([49.25, 49.5, 49.75, 50.25, 50.5,
-                                           50.75]))
+                                           50.75])).astype(float)
         interp_s = ser.reindex(new_index).interpolate(method='pchip')
         # does not blow up, GH5977
         interp_s[49:51]
@@ -908,7 +908,9 @@ def test_interpolate_akima(self):
                           index=Index([1.0, 1.25, 1.5, 1.75,
                                        2.0, 2.25, 2.5, 2.75, 3.0]))
         # interpolate at new_index
-        new_index = ser.index.union(Index([1.25, 1.5, 1.75, 2.25, 2.5, 2.75]))
+        new_index = ser.index.union(
+            Index([1.25, 1.5, 1.75, 2.25, 2.5, 2.75])
+        ).astype(float)
         interp_s = ser.reindex(new_index).interpolate(method='akima')
         assert_series_equal(interp_s[1:3], expected)
 
@@ -921,7 +923,9 @@ def test_interpolate_piecewise_polynomial(self):
                           index=Index([1.0, 1.25, 1.5, 1.75,
                                        2.0, 2.25, 2.5, 2.75, 3.0]))
         # interpolate at new_index
-        new_index = ser.index.union(Index([1.25, 1.5, 1.75, 2.25, 2.5, 2.75]))
+        new_index = ser.index.union(
+            Index([1.25, 1.5, 1.75, 2.25, 2.5, 2.75])
+        ).astype(float)
         interp_s = ser.reindex(new_index).interpolate(
             method='piecewise_polynomial')
         assert_series_equal(interp_s[1:3], expected)
@@ -935,7 +939,9 @@ def test_interpolate_from_derivatives(self):
                           index=Index([1.0, 1.25, 1.5, 1.75,
                                        2.0, 2.25, 2.5, 2.75, 3.0]))
         # interpolate at new_index
-        new_index = ser.index.union(Index([1.25, 1.5, 1.75, 2.25, 2.5, 2.75]))
+        new_index = ser.index.union(
+            Index([1.25, 1.5, 1.75, 2.25, 2.5, 2.75])
+        ).astype(float)
         interp_s = ser.reindex(new_index).interpolate(
             method='from_derivatives')
         assert_series_equal(interp_s[1:3], expected)
diff --git a/pandas/tests/series/test_operators.py b/pandas/tests/series/test_operators.py
index 4d3c9926fc5ae..bbcec0dfe1fd5 100644
--- a/pandas/tests/series/test_operators.py
+++ b/pandas/tests/series/test_operators.py
@@ -14,10 +14,12 @@
 from pandas import (
     Categorical, DataFrame, Index, Series, bdate_range, date_range, isna)
 from pandas.core import ops
+from pandas.core.indexes.base import InvalidIndexError
 import pandas.core.nanops as nanops
 import pandas.util.testing as tm
 from pandas.util.testing import (
-    assert_almost_equal, assert_frame_equal, assert_series_equal)
+    assert_almost_equal, assert_frame_equal, assert_index_equal,
+    assert_series_equal)
 
 from .common import TestData
 
@@ -177,7 +179,6 @@ def test_scalar_na_logical_ops_corners(self):
         operator.and_,
         operator.or_,
         operator.xor,
-
     ])
     def test_logical_ops_with_index(self, op):
         # GH#22092, GH#19792
@@ -196,6 +197,37 @@ def test_logical_ops_with_index(self, op):
         result = op(ser, idx2)
         assert_series_equal(result, expected)
 
+    @pytest.mark.parametrize('op', [
+        pytest.param(ops.rand_,
+                     marks=pytest.mark.xfail(reason="GH#22092 Index "
+                                                    "implementation returns "
+                                                    "Index",
+                                             raises=AssertionError,
+                                             strict=True)),
+        pytest.param(ops.ror_,
+                     marks=pytest.mark.xfail(reason="Index.get_indexer "
+                                                    "with non unique index",
+                                             raises=InvalidIndexError,
+                                             strict=True)),
+        ops.rxor,
+    ])
+    def test_reversed_logical_ops_with_index(self, op):
+        # GH#22092, GH#19792
+        ser = Series([True, True, False, False])
+        idx1 = Index([True, False, True, False])
+        idx2 = Index([1, 0, 1, 0])
+
+        # symmetric_difference is only for rxor, but other 2 should fail
+        expected = idx1.symmetric_difference(ser)
+
+        result = op(ser, idx1)
+        assert_index_equal(result, expected)
+
+        expected = idx2.symmetric_difference(ser)
+
+        result = op(ser, idx2)
+        assert_index_equal(result, expected)
+
     @pytest.mark.parametrize("op, expected", [
         (ops.rand_, pd.Index([False, True])),
         (ops.ror_, pd.Index([False, True])),


Code review comments:
diff_hunk: @@ -0,0 +1,70 @@
+'''
+The tests in this package are to ensure the proper resultant dtypes of
+set operations.
+'''
+import itertools as it
+
+import numpy as np
+import pytest
+
+from pandas.core.dtypes.common import is_dtype_equal
+
+import pandas as pd
+from pandas import Int64Index, RangeIndex
+from pandas.tests.indexes.conftest import indices_list
+import pandas.util.testing as tm
+
+COMPATIBLE_INCONSISTENT_PAIRS = {
+    (Int64Index, RangeIndex): (tm.makeIntIndex, tm.makeRangeIndex)
+}
+
+
+def test_union_same_types(indices):
+    # Union with a non-unique, non-monotonic index raises error
+    # Only needed for bool index factory
+    idx1 = indices.sort_values()
+    idx2 = indices.sort_values()
+    assert idx1.union(idx2).dtype == idx1.dtype
+
+
+@pytest.mark.parametrize(
path: pandas/tests/indexes/test_setops.py
text: @user1:
I think its better if you make a local fixture that makes the pairs, then just xfail the incompatible ones, so the test functions become really simple then and the failing cases are obvious

diff_hunk: @@ -859,6 +859,37 @@ Bug in :func:`DataFrame.to_dict` raises ``ValueError`` when used with
 
     df.to_dict(orient='index')
 
+.. _whatsnew_0240.api_breaking.incompatible_index_unions
+
+Incompatible Index Type Unions
path: doc/source/whatsnew/v0.24.0.rst
text: @user1:
can you move to 0.25.0