[WIP] API/CLN: Refactor DataFrame.append
State: closed

This is a WIP pull request, there are still a bunch of things to do. I'm posting it here so you can give me early feedback or point me in an direction if something is not looking good.

*PR will be split into multiple smaller ones, as suggested*

### How it is

DataFrame.append method is pretty confusing as it is right now. There are multiple codeways according to different input types (dict/Series/DataFrame or list of these) and the results are undefined in lots of edge cases.

### Objective

This PR aims to make the code a bit cleaner and define the behavior for `append` better. For example, the input type should not interfere with the result (i.e. whether `other` is a DataFrame, Series, etc.)

Also, I wanted to make changes to the behavior of `concat` easier. `append` moves its heavy-load to `concat` and, the way it was before, it was hard to understand what was expected from `concat`. Now it's a bit easier.

### Behavior changes

(there might be some things I forgot)

- Accepted inputs for `other` are now exactly `dict`, `Series`, `DataFrame` or a list of any of those. Empty lists are also accepted. Wrong input types produce a `TypeError`

- The behavior for each type of input now is exactly the same for equivalent inputs.

- Empty columns and empty indexes will not be used when calculating the result dtypes. This is something I will think a bit about still, if you have any early suggestions, it will help.

- DataFrames containing any types of columns can be appended now. In contrast, `IntervalIndex`, `PeriodIndex` and `MultiIndex` raised errors before.

- `sort=False` will never sort the resulting columns. `append` used to sort them in some cases.

- ~~`sort=True` will raise a TypeError when the resulting columns cannot be sorted~~ (it may be better to just issue a RuntimeWarning)

- An `InvalidIndexError` will be raised if there are duplicates in the columns indexes and all indexes ~~are not the same (id)~~ do no share the same values (will raise an issue to discuss this)

### Task list (not in order)

- [x] Implement sort=None + regression tests
- [ ] Behavior when sort is not possible (warn (?))
- [ ] Move towards deprecating sort=None
- [x] Behavior for empty indexes
- [ ] Some test cases
- [ ] Check/Improve performance
- [ ] Beautiful PR
- [ ] Write Docs
- [ ] What's new
- [ ] Related issues

Labels:

Comments:
Hello @araraonline! Thanks for updating the PR.

 - There are no PEP8 issues in the file [`pandas/core/frame.py`](https://github.com/araraonline/pandas/blob/4a5494205c6e6cab06053f175d4bf840f8c73424/pandas/core/frame.py) !

 - In the file [`pandas/core/indexes/api.py`](https://github.com/araraonline/pandas/blob/4a5494205c6e6cab06053f175d4bf840f8c73424/pandas/core/indexes/api.py), following                are the PEP8 issues :
> [Line 294:49](https://github.com/araraonline/pandas/blob/4a5494205c6e6cab06053f175d4bf840f8c73424/pandas/core/indexes/api.py#L294): [E128](https://duckduckgo.com/?q=pep8%20E128) continuation line under-indented for visual indent
> [Line 299:39](https://github.com/araraonline/pandas/blob/4a5494205c6e6cab06053f175d4bf840f8c73424/pandas/core/indexes/api.py#L299): [E128](https://duckduckgo.com/?q=pep8%20E128) continuation line under-indented for visual indent


 - There are no PEP8 issues in the file [`pandas/core/indexes/base.py`](https://github.com/araraonline/pandas/blob/4a5494205c6e6cab06053f175d4bf840f8c73424/pandas/core/indexes/base.py) !

 - In the file [`pandas/tests/reshape/test_append.py`](https://github.com/araraonline/pandas/blob/4a5494205c6e6cab06053f175d4bf840f8c73424/pandas/tests/reshape/test_append.py), following                are the PEP8 issues :
> [Line 79:6](https://github.com/araraonline/pandas/blob/4a5494205c6e6cab06053f175d4bf840f8c73424/pandas/tests/reshape/test_append.py#L79): [E121](https://duckduckgo.com/?q=pep8%20E121) continuation line under-indented for hanging indent
> [Line 84:6](https://github.com/araraonline/pandas/blob/4a5494205c6e6cab06053f175d4bf840f8c73424/pandas/tests/reshape/test_append.py#L84): [E121](https://duckduckgo.com/?q=pep8%20E121) continuation line under-indented for hanging indent
> [Line 866:17](https://github.com/araraonline/pandas/blob/4a5494205c6e6cab06053f175d4bf840f8c73424/pandas/tests/reshape/test_append.py#L866): [E127](https://duckduckgo.com/?q=pep8%20E127) continuation line over-indented for visual indent
> [Line 867:32](https://github.com/araraonline/pandas/blob/4a5494205c6e6cab06053f175d4bf840f8c73424/pandas/tests/reshape/test_append.py#L867): [E721](https://duckduckgo.com/?q=pep8%20E721) do not compare types, use 'isinstance()'
> [Line 991:80](https://github.com/araraonline/pandas/blob/4a5494205c6e6cab06053f175d4bf840f8c73424/pandas/tests/reshape/test_append.py#L991): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (80 > 79 characters)


 - There are no PEP8 issues in the file [`pandas/tests/reshape/test_concat.py`](https://github.com/araraonline/pandas/blob/4a5494205c6e6cab06053f175d4bf840f8c73424/pandas/tests/reshape/test_concat.py) !



##### Comment last updated on October 02, 2018 at 01:18 Hours UTC
I haven't had a chance to look yet, but FYI this will be very hard to review in its current state.

If possible, I'd recommend identifying small, self-contained changes that can be split off into smaller pull requests. If you're changing anything substantially, it's best to open an issue to discuss API changes and design, before investing too much time in the changes.
Oh thanks, I first thought of rebasing this into multiple small commits that would be explanatory, but splitting into PR's look better. I will also keep in mind the API changes
@TomAugspurger @jreback 

What way would be easier to review if I'm thinking of making various progressive changes in the code? Some thoughts:

1. PR everything together explaining changes through commits
2. Work locally on branches that depend of one another, send PR one at a time
3. Work locally on branches that depend of one another, send all PR's at once
Probably 2 or 3, but I haven't had time to go through #22957 in detail.

You're timing here is just a *bit* to early :) We're in the middle of a largish refactor to our data types

- sparse: https://github.com/pandas-dev/pandas/pull/22325
- period: https://github.com/pandas-dev/pandas/pull/22862
- concat on EAs: https://github.com/pandas-dev/pandas/pull/22997

After that, we'll have one more for Datetimes and Timedeltas, and then the dust will settle a bit. There's a decent chance that sparse and period are done by next week, and datetimes / Timedeltas a little after that (a week or two).

I *suspect* that all those changes will fix the issues you saw with Period, Datetime, Timedelta, and Interval indexes.

So maybe see if there are any fixes you can make for MultiIndex that are relatively standalone?
Hmmm... Sure! Didn't notice there was so much going on right now.

Think I will wait a little bit or focus on the MultiIndex as suggested
closing as stale. if you'd like to continue, pls ping.