BUG: fix union_indexes not supporting sort=False for Index subclasses
State: closed

- [X] closes #35092
- [X] 1 tests added / 1 passed
- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [X] whatsnew entry

# Scope of PR
The reason for `DataFrame.append` unnecessarily sorting columns turned out to be that when merging indexes we were not passing the `sort` argument for an `Index` subclass. This PR fixes this bug.

# Details
At some point down the call stack, `DataFrame.append` calls `union_indexes` in `core.indexes.api`. In it we detect the type of our indexes: 
```python
indexes, kind = _sanitize_and_check(indexes)
```
We don't pass `sort` for `kind == 'special'` in `union_indexes`.

I had some other ideas, but what I ended up doing is to simply pass the `sort` argument to `Index.union` for `kind == 'special'`. We do have to ignore the `sort` argument for `[MultiIndex, RangeIndex, DatetimeIndex, CategoricalIndex]` to avoid breaking tests and backward compatibility, but it doesn't make sense for these ones to not be sorted when joined anyway, in my opinion.

# Performance
There are no changes to algorithms called. I still did some benchmarking to compare with my previous approach. The current solution didn't have any negative performance impact.


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
Hm. There are side-effects, it seems. When the first index is a subset of the second, the extra elements of the second come in their original order now. I'll research this tomorrow.
Researching. Fiddling with `pd.concat` and `union_indexes` affects a bunch of stuff. Let's see if I can get this bugfix to work without introducing undesired behavior elsewhere.
The current idea is to support `sort` only for `Index` subclasses that are just an `Index` of a specific datatype.

Update: okay, seems to work. I'll see if I can benchmark tomorrow, and it should be ready for a review.
Update 2: benchmarked it, there were some decreases, so I did some more digging and ended up simply passing `sort` to `Index.union` instead of all the Index subclass detection.
@gfyoung This PR is ready for a review now.
Web and docs seem to be failing for everyone atm.
CI / Checks now fail because `isort` dropped the `--recursive` key. Submitted an issue #35134 
Hello @AlexKirko! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:













There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-07-08 16:02:22 UTC
@jreback Changes made, ready for another review:
1. Pass sort through to all `Index` subclasses. Alter tests to accomodate no automatic sorting. Add comments to changes where appropriate. After another look, I don't think this change will break anything for our users.
2. Add test for `append` to `reshape/test_concat.py`. Use streamlined equivalent of the OP's test case, as the original case is overly verbose.
3. Move whatsnew record to reshape section and remove mention of the private function.
All green.
@jreback 
All green. There was some hardware incident that impacted Azure pipelines, but we are good now.
thanks very nice!