Repository: pandas-dev/pandas
Pull request: BUG/API: Index.append with mixed object/Categorical indices
Closes #14298, related to #13660 and #13767

This closes the `info()` issue with CategoricalIndex, but the underlying issue is actually related to Index.append, when appending mixed dtype index objects: 

```
In [11]: pd.Index(['a']).append(pd.CategoricalIndex(['a', 'b']))
...
AttributeError: 'Index' object has no attribute '_is_dtype_compat'
```

This error occurs when the calling index is not a Categorical (because of https://github.com/pandas-dev/pandas/blob/v0.19.0/pandas/indexes/base.py#L1439 the Categorical `_is_dtype_compat` method is used anyway).

But the question is, what should the result be of the above expression?

- a `CategoricalIndex` ? -> IMO not, as the calling index is not a CategoricalIndex, the result should also not be one (regardless of what is appended) -> so the intention of the code is IMO also not correct
- an object Index ? -> yes, this is also what 0.18.1 does

But, the more important discussion is *what to do when the calling index is a CategoricalIndex* ? In 0.18 (and in 0.19.0) this either returned a CategoricalIndex or raised an error:

```
In [13]: pd.__version__
Out[13]: u'0.18.1'

In [104]: pd.CategoricalIndex(['a', 'b']).append(pd.Index(['a']))
Out[104]: CategoricalIndex([u'a', u'b', u'a'], categories=[u'a', u'b'], ordered=False, dtype='category')

In [105]: pd.CategoricalIndex(['a', 'b']).append(pd.Index(['c']))
...
TypeError: cannot append a non-category item to a CategoricalIndex
```

But, for the `concat` case we decided to return an object dtyped Index in those cases instead of coercing or raising (see discussion in #13767 and the summary table with rules in https://github.com/pandas-dev/pandas/pull/13767#issuecomment-237806603). So the question is, do think that `append` should follow the same rules as `concat`. Or, do we want to be more flexible for `append`, and let it depend on the type of the calling Index?


cc @sinhrks 


Problem statement:
.info() fails with CategoricalIndex in the columns
Just in the columns, the index doesn't cause any issues. This came up as the output of `pd.get_dummies` can have a CategoricalIndex in the columns.
#### A small, complete example of the issue

``` python
In [31]: idx = pd.CategoricalIndex(['a', 'b'])

In [32]: df = pd.DataFrame(np.zeros((2, 2)), index=idx, columns=idx)

In [33]: df.info()
```

``` pytb
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-33-857958930b27> in <module>()
----> 1 df.info()

/Users/tom.augspurger/Envs/py3/lib/python3.5/site-packages/pandas-0.19.0rc1+21.ge596cbf-py3.5-macosx-10.11-x86_64.egg/pandas/core/frame.py in info(self, verbose, buf, max_cols, memory_usage, null_counts)
   1790                 if 'object' in counts or is_object_dtype(self.index):
   1791                     size_qualifier = '+'
-> 1792             mem_usage = self.memory_usage(index=True, deep=deep).sum()
   1793             lines.append("memory usage: %s\n" %
   1794                          _sizeof_fmt(mem_usage, size_qualifier))

/Users/tom.augspurger/Envs/py3/lib/python3.5/site-packages/pandas-0.19.0rc1+21.ge596cbf-py3.5-macosx-10.11-x86_64.egg/pandas/core/frame.py in memory_usage(self, index, deep)
   1827         if index:
   1828             result = Series(self.index.memory_usage(deep=deep),
-> 1829                             index=['Index']).append(result)
   1830         return result
   1831

/Users/tom.augspurger/Envs/py3/lib/python3.5/site-packages/pandas-0.19.0rc1+21.ge596cbf-py3.5-macosx-10.11-x86_64.egg/pandas/core/series.py in append(self, to_append, ignore_index, verify_integrity)
   1593             to_concat = [self, to_append]
   1594         return concat(to_concat, ignore_index=ignore_index,
-> 1595                       verify_integrity=verify_integrity)
   1596
   1597     def _binop(self, other, func, level=None, fill_value=None):

/Users/tom.augspurger/Envs/py3/lib/python3.5/site-packages/pandas-0.19.0rc1+21.ge596cbf-py3.5-macosx-10.11-x86_64.egg/pandas/tools/merge.py in concat(objs, axis, join, join_axes, ignore_index, keys, levels, names, verify_integrity, copy)
   1323                        keys=keys, levels=levels, names=names,
   1324                        verify_integrity=verify_integrity,
-> 1325                        copy=copy)
   1326     return op.get_result()
   1327

/Users/tom.augspurger/Envs/py3/lib/python3.5/site-packages/pandas-0.19.0rc1+21.ge596cbf-py3.5-macosx-10.11-x86_64.egg/pandas/tools/merge.py in __init__(self, objs, axis, join, join_axes, keys, levels, names, ignore_index, verify_integrity, copy)
   1462         self.copy = copy
   1463
-> 1464         self.new_axes = self._get_new_axes()
   1465
   1466     def get_result(self):

/Users/tom.augspurger/Envs/py3/lib/python3.5/site-packages/pandas-0.19.0rc1+21.ge596cbf-py3.5-macosx-10.11-x86_64.egg/pandas/tools/merge.py in _get_new_axes(self)
   1550                 new_axes[i] = ax
   1551
-> 1552         new_axes[self.axis] = self._get_concat_axis()
   1553         return new_axes
   1554

/Users/tom.augspurger/Envs/py3/lib/python3.5/site-packages/pandas-0.19.0rc1+21.ge596cbf-py3.5-macosx-10.11-x86_64.egg/pandas/tools/merge.py in _get_concat_axis(self)
   1604
   1605         if self.keys is None:
-> 1606             concat_axis = _concat_indexes(indexes)
   1607         else:
   1608             concat_axis = _make_concat_multiindex(indexes, self.keys,

/Users/tom.augspurger/Envs/py3/lib/python3.5/site-packages/pandas-0.19.0rc1+21.ge596cbf-py3.5-macosx-10.11-x86_64.egg/pandas/tools/merge.py in _concat_indexes(indexes)
   1622
   1623 def _concat_indexes(indexes):
-> 1624     return indexes[0].append(indexes[1:])
   1625
   1626

/Users/tom.augspurger/Envs/py3/lib/python3.5/site-packages/pandas-0.19.0rc1+21.ge596cbf-py3.5-macosx-10.11-x86_64.egg/pandas/indexes/base.py in append(self, other)
   1425             # if any of the to_concat is category
   1426             from pandas.indexes.category import CategoricalIndex
-> 1427             return CategoricalIndex._append_same_dtype(self, to_concat, name)
   1428
   1429         if len(typs) == 1:

/Users/tom.augspurger/Envs/py3/lib/python3.5/site-packages/pandas-0.19.0rc1+21.ge596cbf-py3.5-macosx-10.11-x86_64.egg/pandas/indexes/category.py in _append_same_dtype(self, to_concat, name)
    578         ValueError if other is not in the categories
    579         """
--> 580         to_concat = [self._is_dtype_compat(c) for c in to_concat]
    581         codes = np.concatenate([c.codes for c in to_concat])
    582         result = self._create_from_codes(codes, name=name)

/Users/tom.augspurger/Envs/py3/lib/python3.5/site-packages/pandas-0.19.0rc1+21.ge596cbf-py3.5-macosx-10.11-x86_64.egg/pandas/indexes/category.py in <listcomp>(.0)
    578         ValueError if other is not in the categories
    579         """
--> 580         to_concat = [self._is_dtype_compat(c) for c in to_concat]
    581         codes = np.concatenate([c.codes for c in to_concat])
    582         result = self._create_from_codes(codes, name=name)

AttributeError: 'Index' object has no attribute '_is_dtype_compat'
```
#### Expected Output

`df.info()`
#### Output of `pd.show_versions()`

<details>
In [34]: pd.show_versions()
## INSTALLED VERSIONS

commit: None
python: 3.5.2.final.0
python-bits: 64
OS: Darwin
OS-release: 15.6.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.19.0rc1+21.ge596cbf
nose: 1.3.7
pip: 8.1.2
setuptools: 26.1.1
Cython: 0.25a0
numpy: 1.11.1
scipy: 0.18.0
statsmodels: 0.8.0rc1
xarray: 0.7.2
IPython: 5.1.0
sphinx: 1.4.6
patsy: 0.4.1
dateutil: 2.5.3
pytz: 2016.6.1
blosc: 1.4.1
bottleneck: 1.0.0
tables: 3.2.2
numexpr: 2.6.1
matplotlib: 1.5.2
openpyxl: 2.3.5
xlrd: 0.9.4
xlwt: 1.0.0
xlsxwriter: None
lxml: 3.4.4
bs4: 4.4.1
html5lib: 0.9999999
httplib2: 0.9.2
apiclient: 1.5.1
sqlalchemy: 1.0.12
pymysql: 0.7.6.None
psycopg2: 2.6.2 (dt dec pq3 ext lo64)
jinja2: 2.8
boto: 2.39.0
pandas_datareader: None
</details>

Hints from issue discussion:
Similar example from #14450 

```
df = pd.DataFrame({"data": np.arange(0, 10, 0.1)})
v = pd.cut(df.data, [0, 1, 2, 5, 10], include_lowest=True).rename("cuts")
df.join(pd.get_dummies(v))
```

Commit message: BUG/API: Index.append with mixed object/Categorical indices

Patch to review:
diff --git a/pandas/indexes/base.py b/pandas/indexes/base.py
index 4d2dcd259e623..883076542cf88 100644
--- a/pandas/indexes/base.py
+++ b/pandas/indexes/base.py
@@ -1466,11 +1466,6 @@ def append(self, other):
 
         typs = _concat.get_dtype_kinds(to_concat)
 
-        if 'category' in typs:
-            # if any of the to_concat is category
-            from pandas.indexes.category import CategoricalIndex
-            return CategoricalIndex._append_same_dtype(self, to_concat, name)
-
         if len(typs) == 1:
             return self._append_same_dtype(to_concat, name=name)
         return _concat._concat_index_asobject(to_concat, name=name)
diff --git a/pandas/tests/indexes/test_category.py b/pandas/tests/indexes/test_category.py
index 9f8405bcc2e1e..4f2d41f86e6c3 100644
--- a/pandas/tests/indexes/test_category.py
+++ b/pandas/tests/indexes/test_category.py
@@ -272,11 +272,17 @@ def test_append(self):
 
         # with objects
         result = ci.append(Index(['c', 'a']))
-        expected = CategoricalIndex(list('aabbcaca'), categories=categories)
+        #expected = CategoricalIndex(list('aabbcaca'), categories=categories)
+        expected = Index(list('aabbcaca'))
         tm.assert_index_equal(result, expected, exact=True)
 
         # invalid objects
-        self.assertRaises(TypeError, lambda: ci.append(Index(['a', 'd'])))
+        #self.assertRaises(TypeError, lambda: ci.append(Index(['a', 'd'])))
+
+        # GH14298 - if base object is not categorical -> coerce to object
+        result = Index(['c', 'a']).append(ci)
+        expected = Index(list('caaabbca'))
+        tm.assert_index_equal(result, expected, exact=True)
 
     def test_insert(self):
 


Code review comments:
diff_hunk: @@ -1464,13 +1464,13 @@ def append(self, other):
         names = set([obj.name for obj in to_concat])
         name = None if len(names) > 1 else self.name
 
-        typs = _concat.get_dtype_kinds(to_concat)
-
-        if 'category' in typs:
+        if self.is_categorical():
             # if any of the to_concat is category
path: pandas/indexes/base.py
text: @user1:
then you should change this comment as this is no longer true (the Index must be a CI to return a CI) only.