ENH: DataFrame.append preserves columns dtype if possible
State: closed

- [x] closes #18359
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

This  PR makes ``DataFrame.append`` preserve columns dtype, e.g.:

```python
>>> idx = pd.CategoricalIndex('a b'.split())
>>> df = pd.DataFrame([[1, 2]], columns=idx)
>>> ser = pd.Series([3, 4], index=idx, name=1)
>>> df.append(ser).columns
CategoricalIndex(['a', 'b'], categories=['a', 'b'], ordered=False, dtype='category')
```

Previously, the above returned ``Index(['a', 'b'], dtype='object')``, i.e. the index type information was lost when using ``append``.

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
Can't get Travis-ci to complete because it uses too much time, somehow.

I assume this has nothing to do with this PR, but is about travis settings?
rebase and push again, fixed some hanging by Travis CI
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/19021?src=pr&el=h1) Report
> Merging [#19021](https://codecov.io/gh/pandas-dev/pandas/pull/19021?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/bb095a6e96217f162544b10e9e7a46f04071fb37?src=pr&el=desc) will **decrease** coverage by `0.02%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/19021/graphs/tree.svg?width=650&src=pr&token=eZ4WkYLtcO&height=150)](https://codecov.io/gh/pandas-dev/pandas/pull/19021?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #19021      +/-   ##
==========================================
- Coverage   91.84%   91.82%   -0.03%     
==========================================
  Files         153      153              
  Lines       49295    49299       +4     
==========================================
- Hits        45276    45268       -8     
- Misses       4019     4031      +12
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.21% <ø> (-0.03%)` | :arrow_down: |
| #single | `41.89% <ø> (-0.01%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/19021?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/19021/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.16% <ø> (ø)` | :arrow_up: |
| [pandas/plotting/\_converter.py](https://codecov.io/gh/pandas-dev/pandas/pull/19021/diff?src=pr&el=tree#diff-cGFuZGFzL3Bsb3R0aW5nL19jb252ZXJ0ZXIucHk=) | `65.07% <0%> (-1.74%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/19021?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/19021?src=pr&el=footer). Last update [bb095a6...13537ae](https://codecov.io/gh/pandas-dev/pandas/pull/19021?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

Ok, rebased.
Changed and green (except for travis errors that seem unrelated).

A different subject, but shouldn't ``.reindex`` be able to align ``Interval`` and ``int`` objects?
Hello @topper-123! Thanks for updating the PR.







Cheers ! There are no PEP8 issues in this Pull Request. :beers: 

##### Comment last updated on April 17, 2018 at 17:28 Hours UTC
All is green again, now that #19112 has been pulled in.
@topper-123 added some ids to the parametrization. also I added a test case in the combo test or an ordered and unordered category which is failing. (its commented out). can you have a look?
also pls rebase
 @jreback , wrt. the ordered vs unordered combo in ``test_append_different_columns_types``: ``pd.CategoricalIndex('A B C'.split())`` and ``pd.CategoricalIndex('A B C'.split(), ordered=True)`` combine into ``pd.CategoricalIndex('A B C'.split())``. To pass this test, the combined columns have to have length 6, but this has length 3.

I think the current implementation is reasonable behaviour, as ``append`` tries to find the lowest common denominator, which in this case is ``pd.CategoricalIndex('A B C'.split())``. Possibly you could give append a ``strict`` parameter where you'd raise if the two dtypes aren't the same, but that's another issue than this one, IMO.
cc @jschendel 
I've looked into if ``df.append(ser)`` should work, but I think this is beyond me, sorry.

The issue seems to be that ``IntervalIndex.get_loc`` is ambigous wrt. what input means:

```python
>> ii = pd.IntervalIndex.from_breaks([0,1,2,3])
>> i = pd.Index([1, 2])
>> combined = ii.append(i)
>> combined
Index([(0, 1], (1, 2], (2, 3], 0, 1, 2], dtype='object')  # so far so good
```
Now, to proceed, we have to call ``ii.get_indexer(combined)`` which in turn calls ``ii.get_loc(x)`` for each value in ``combined`` and we should get:

```python
>> ii.get_indexer(combined)
[0, 1, 2, -1, -1, -1]  # what we should get
[0, 1, 2, -1, 0, 1]  # what we're actually getting
```
The reason for this bug is that ``ii.get_loc(1)`` evaluates as 1 for for ``ii``, because 1 is in the interval (1, 2], but should for the purpose of this indexer give -1.

All in all, I think this is too deep for me to fix, sorry. I'll report an issue, and hopefully someone can take it on.

Can this PR be merged regardless, maybe with a xfail for ``IntervalIndex``
I've addressed the comments, except one where I've commented specifically.

If #19353 (exact matches for CategoricalIndex) or a version thereof could be accepted, it would be possible to append frames/series with CategoricalIndex as well, i.e. move IntervalIndex checks from ``test_append_different_columns_types_raises`` to ``test_append_different_columns_types``.
I've uploaded a version with fixture, but honestly I consider this less readable than the previous version...
can you update
can you rebase
@topper-123 can you rebase
Rebased.

I've also simplified the tests a bit. I dropped the fixture-based approach, as I don't think it makes things clearer in this case.
do you disagree wit the approach, @jreback?
@jreback, could you pull this in?
thanks @topper-123 
A related question.

I was expecting making a column an index , or level in  a MultiIndex, to not change the dtype.

```python
df  = pd.DataFrame(dict(x=np.array([1,2],dtype = 'int32'), 
                        y =['a', 'b']))

df.set_index('x').reset_index().info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 2 entries, 0 to 1
Data columns (total 2 columns):
x    2 non-null int64
y    2 non-null object
dtypes: int64(1), object(1)
memory usage: 104.0+ bytes
```
Is this exepected ,and will this enhancement change this?  Thanks 

@jonblunt 

The index will in this case be a Int64Index:

```python
>>> df.set_index('x').index
Int64Index([1, 2], dtype='int64', name='x')
```

Pandas doesn't have a Int32Index, so this behaviour is as expected and you shouldn't expect an int32 when resetting (or you'll have to reset manually).