Repository: pandas-dev/pandas

Issue:
BUG: Index.difference and Index.intersection doesn't preserve type of Index for some Index subclasses for corner cases
#### Code Sample, a copy-pastable example if possible

```python
pi1 = pd.PeriodIndex(start='2000', end='2010', freq='A')
print(pi1.difference(pi1), pi1.intersection(pi1.drop(pi1)))

ci = pd.CategoricalIndex(['a','b','c'], categories=['a','b','c'])
print(ci.difference(ci), ci.intersection(ci.drop(ci)))

ri = pd.RangeIndex(start=1, stop=5)
print(ri.difference(ri), ri.intersection(ri.drop(ri)))

```
#### Problem description
The result of taking the difference of an `Index` for various `Index` subclasses and the `Index` produces a resulting `Index` that does not preserve the type of the subclass.

From a set algebra point of view, for a set S, `S.difference(S)` should equal `S.intersection(nullset)`.

The output from the above is:
```
Index([], dtype='object') PeriodIndex([], dtype='period[A-DEC]', freq='A-DEC')
Index([], dtype='object') CategoricalIndex([], categories=['a', 'b', 'c'], ordered=False, dtype='category')
Index([], dtype='object') Int64Index([], dtype='int64')
```

There is some discussion in the pull request #19849, where I discovered this bug, but at request of @jreback, I have split this into a separate issue.

#### Expected Output
```
PeriodIndex([], dtype='period[A-DEC]', freq='A-DEC') PeriodIndex([], dtype='period[A-DEC]', freq='A-DEC')
CategoricalIndex([], categories=['a', 'b', 'c'], ordered=False, dtype='category') CategoricalIndex([], categories=['a', 'b', 'c'], ordered=False, dtype='category')
RangeIndex(start=0, stop=0, step=1) RangeIndex(start=0, stop=0, step=1)
```

Note that for `RangeIndex`, the result of the `intersection` operation is also incorrect.
#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit: None
python: 3.6.4.final.0
python-bits: 64
OS: Windows
OS-release: 10
machine: AMD64
processor: Intel64 Family 6 Model 60 Stepping 3, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.None

pandas: 0.22.0
pytest: 3.3.2
pip: 9.0.1
setuptools: 38.4.0
Cython: 0.27.3
numpy: 1.14.0
scipy: 1.0.0
pyarrow: None
xarray: None
IPython: 6.2.1
sphinx: 1.6.6
patsy: 0.5.0
dateutil: 2.6.1
pytz: 2017.3
blosc: None
bottleneck: 1.2.1
tables: 3.4.2
numexpr: 2.6.4
feather: None
matplotlib: 2.1.2
openpyxl: 2.4.10
xlrd: 1.1.0
xlwt: 1.3.0
xlsxwriter: 1.0.2
lxml: 4.1.1
bs4: 4.6.0
html5lib: 1.0.1
sqlalchemy: 1.2.1
pymysql: 0.7.11.None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
</details>



Discussion:
I'm willing to work on this, but can we have a discussion on the implementation?  The suggested solution in the discussion in #19849 is to use `self._shallow_copy([])`, but that method doesn't work right for empty indexes, so I think it is easier to just have a method that creates an empty index, but preserves the other properties of the index (e.g., categories for `CategoricalIndex`, range step for `RangeIndex`, freq for `PeriodIndex`, etc.)

Alternatively, I can make `self._shallow_copy([])` work for the various `Index` subclasses with an empty list argument.
@Dr-Irv : That seems like a good first attempt to patch this, though other options are welcome of course.
@gfyoung By "That seems", do you mean having a method to create an empty index, or fixing `_shallow_copy([])`
Oh, sorry!  I was referring to fixing `_shallow_copy([])`.

Resolving patch:
diff --git a/pandas/core/indexes/base.py b/pandas/core/indexes/base.py
--- a/pandas/core/indexes/base.py
+++ b/pandas/core/indexes/base.py
@@ -457,7 +457,7 @@ def _simple_new(cls, values, name=None, dtype=None, **kwargs):
         Must be careful not to recurse.
         """
         if not hasattr(values, 'dtype'):
-            if values is None and dtype is not None:
+            if (values is None or not len(values)) and dtype is not None:
                 values = np.empty(0, dtype=dtype)
             else:
                 values = np.array(values, copy=False)
@@ -491,6 +491,8 @@ def _shallow_copy(self, values=None, **kwargs):
             values = self.values
         attributes = self._get_attributes_dict()
         attributes.update(kwargs)
+        if not len(values) and 'dtype' not in kwargs:
+            attributes['dtype'] = self.dtype
         return self._simple_new(values, **attributes)
 
     def _shallow_copy_with_infer(self, values=None, **kwargs):
@@ -511,6 +513,8 @@ def _shallow_copy_with_infer(self, values=None, **kwargs):
         attributes = self._get_attributes_dict()
         attributes.update(kwargs)
         attributes['copy'] = False
+        if not len(values) and 'dtype' not in kwargs:
+            attributes['dtype'] = self.dtype
         if self._infer_as_myclass:
             try:
                 return self._constructor(values, **attributes)
@@ -2815,7 +2819,7 @@ def difference(self, other):
         self._assert_can_do_setop(other)
 
         if self.equals(other):
-            return Index([], name=self.name)
+            return self._shallow_copy([])
 
         other, result_name = self._convert_can_do_setop(other)
 
diff --git a/pandas/core/indexes/multi.py b/pandas/core/indexes/multi.py
--- a/pandas/core/indexes/multi.py
+++ b/pandas/core/indexes/multi.py
@@ -2755,7 +2755,7 @@ def intersection(self, other):
         other_tuples = other._ndarray_values
         uniq_tuples = sorted(set(self_tuples) & set(other_tuples))
         if len(uniq_tuples) == 0:
-            return MultiIndex(levels=[[]] * self.nlevels,
+            return MultiIndex(levels=self.levels,
                               labels=[[]] * self.nlevels,
                               names=result_names, verify_integrity=False)
         else:
@@ -2777,7 +2777,7 @@ def difference(self, other):
             return self
 
         if self.equals(other):
-            return MultiIndex(levels=[[]] * self.nlevels,
+            return MultiIndex(levels=self.levels,
                               labels=[[]] * self.nlevels,
                               names=result_names, verify_integrity=False)
 


Test patch:
diff --git a/pandas/tests/indexes/test_base.py b/pandas/tests/indexes/test_base.py
--- a/pandas/tests/indexes/test_base.py
+++ b/pandas/tests/indexes/test_base.py
@@ -20,7 +20,7 @@
 from pandas import (period_range, date_range, Series,
                     DataFrame, Float64Index, Int64Index, UInt64Index,
                     CategoricalIndex, DatetimeIndex, TimedeltaIndex,
-                    PeriodIndex, isna)
+                    PeriodIndex, RangeIndex, isna)
 from pandas.core.index import _get_combined_index, _ensure_index_from_sequences
 from pandas.util.testing import assert_almost_equal
 from pandas.compat.numpy import np_datetime64_compat
@@ -44,7 +44,7 @@ def setup_method(self, method):
                             tdIndex=tm.makeTimedeltaIndex(100),
                             intIndex=tm.makeIntIndex(100),
                             uintIndex=tm.makeUIntIndex(100),
-                            rangeIndex=tm.makeIntIndex(100),
+                            rangeIndex=tm.makeRangeIndex(100),
                             floatIndex=tm.makeFloatIndex(100),
                             boolIndex=Index([True, False]),
                             catIndex=tm.makeCategoricalIndex(100),
@@ -57,6 +57,15 @@ def setup_method(self, method):
     def create_index(self):
         return Index(list('abcde'))
 
+    def generate_index_types(self, skip_index_keys=[]):
+        """
+        Return a generator of the various index types, leaving
+        out the ones with a key in skip_index_keys
+        """
+        for key, idx in self.indices.items():
+            if key not in skip_index_keys:
+                yield key, idx
+
     def test_new_axis(self):
         new_index = self.dateIndex[None, :]
         assert new_index.ndim == 2
@@ -406,6 +415,27 @@ def test_constructor_dtypes_timedelta(self):
                         pd.TimedeltaIndex(list(values), dtype=dtype)]:
                 tm.assert_index_equal(res, idx)
 
+    def test_constructor_empty(self):
+        skip_index_keys = ["repeats", "periodIndex", "rangeIndex",
+                           "tuples"]
+        for key, idx in self.generate_index_types(skip_index_keys):
+            empty = idx.__class__([])
+            assert isinstance(empty, idx.__class__)
+            assert not len(empty)
+
+        empty = PeriodIndex([], freq='B')
+        assert isinstance(empty, PeriodIndex)
+        assert not len(empty)
+
+        empty = RangeIndex(step=1)
+        assert isinstance(empty, pd.RangeIndex)
+        assert not len(empty)
+
+        empty = MultiIndex(levels=[[1, 2], ['blue', 'red']],
+                           labels=[[], []])
+        assert isinstance(empty, MultiIndex)
+        assert not len(empty)
+
     def test_view_with_args(self):
 
         restricted = ['unicodeIndex', 'strIndex', 'catIndex', 'boolIndex',
@@ -1034,6 +1064,27 @@ def test_symmetric_difference(self):
         assert tm.equalContents(result, expected)
         assert result.name == 'new_name'
 
+    def test_difference_type(self):
+        # GH 20040
+        # If taking difference of a set and itself, it
+        # needs to preserve the type of the index
+        skip_index_keys = ['repeats']
+        for key, idx in self.generate_index_types(skip_index_keys):
+            result = idx.difference(idx)
+            expected = idx.drop(idx)
+            tm.assert_index_equal(result, expected)
+
+    def test_intersection_difference(self):
+        # GH 20040
+        # Test that the intersection of an index with an
+        # empty index produces the same index as the difference
+        # of an index with itself.  Test for all types
+        skip_index_keys = ['repeats']
+        for key, idx in self.generate_index_types(skip_index_keys):
+            inter = idx.intersection(idx.drop(idx))
+            diff = idx.difference(idx)
+            tm.assert_index_equal(inter, diff)
+
     def test_is_numeric(self):
         assert not self.dateIndex.is_numeric()
         assert not self.strIndex.is_numeric()