ENH: Add sort parameter to set operations for some Indexes and adjust…
State: closed

… tests

- [ ] Progress towards #24471
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

This PR makes some progress towards adding a ``sort`` parameter with a default of ``True`` to the set operations (``union``, ``intersection``, ``difference`` and ``symmetric_difference``) on ``Index`` classes. Adding this parameter into every type of ``Index`` would result in a very big PR so I have decided to break it up and try to add it in stages. I have tried to focus on ``Index``, ``MultiIndex``,  ``PeriodIndex`` and ``Intervalndex`` in this PR but have made some very small changes to set operations on other indices where necessary.

Some issues to consider:
- I'm not sure whether it will be possible to control the sorting behaviour of the results of all of the set operations for all of the ``Index`` types. For example, because of the way some of the set operations are implemented on some ``Index`` types the results may always be sorted even if ``sort=False`` is passed (e.g., a ``union`` operation on ``DatetimeIndex``s may always return a sorted result in some cases even if ``sort=False``). Perhaps this is not really a problem as long as we document it.
- There are some other corner cases that always ignore the ``sort`` parameter at the moment. For example, the ``intersection`` of an unsorted ``Index`` with itself will return the original unsorted ``Index`` even if ``sort=True`` is passed because the ``sort`` parameter is simply ignored in this case. Similarly, the ``intersection`` of an unsorted ``Index`` with an empty ``Index`` will also return the original unsorted ``Index`` even if ``sort=True``. There is similar behaviour for the other set operations.

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: API Design

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24521?src=pr&el=h1) Report
> Merging [#24521](https://codecov.io/gh/pandas-dev/pandas/pull/24521?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/bf9d41c8f211fdd9e2b1a37162b753d3845a6708?src=pr&el=desc) will **increase** coverage by `0.01%`.
> The diff coverage is `23.25%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/24521/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/24521?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #24521      +/-   ##
==========================================
+ Coverage   31.88%    31.9%   +0.01%     
==========================================
  Files         166      166              
  Lines       52434    52444      +10     
==========================================
+ Hits        16718    16730      +12     
+ Misses      35716    35714       -2
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `30.28% <16.27%> (-0.01%)` | :arrow_down: |
| #single | `31.9% <23.25%> (+0.01%)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/24521?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/indexes/interval.py](https://codecov.io/gh/pandas-dev/pandas/pull/24521/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9pbnRlcnZhbC5weQ==) | `35.03% <0%> (+0.12%)` | :arrow_up: |
| [pandas/core/indexes/api.py](https://codecov.io/gh/pandas-dev/pandas/pull/24521/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9hcGkucHk=) | `78.21% <0%> (ø)` | :arrow_up: |
| [pandas/core/indexes/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/24521/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9iYXNlLnB5) | `47.46% <20.68%> (+0.44%)` | :arrow_up: |
| [pandas/core/indexes/range.py](https://codecov.io/gh/pandas-dev/pandas/pull/24521/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9yYW5nZS5weQ==) | `48.97% <25%> (-0.29%)` | :arrow_down: |
| [pandas/core/indexes/datetimes.py](https://codecov.io/gh/pandas-dev/pandas/pull/24521/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9kYXRldGltZXMucHk=) | `37.33% <33.33%> (ø)` | :arrow_up: |
| [pandas/core/indexes/multi.py](https://codecov.io/gh/pandas-dev/pandas/pull/24521/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9tdWx0aS5weQ==) | `30.58% <40%> (-0.05%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24521?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24521?src=pr&el=footer). Last update [bf9d41c...4b31df2](https://codecov.io/gh/pandas-dev/pandas/pull/24521?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24521?src=pr&el=h1) Report
> Merging [#24521](https://codecov.io/gh/pandas-dev/pandas/pull/24521?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/aa05e542bc47140123157c5854cac498c3541a8c?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `94.59%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/24521/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/24521?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #24521      +/-   ##
==========================================
- Coverage   92.38%   92.38%   -0.01%     
==========================================
  Files         166      166              
  Lines       52382    52392      +10     
==========================================
+ Hits        48395    48404       +9     
- Misses       3987     3988       +1
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.81% <91.89%> (ø)` | :arrow_up: |
| #single | `42.91% <45.94%> (ø)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/24521?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/indexes/multi.py](https://codecov.io/gh/pandas-dev/pandas/pull/24521/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9tdWx0aS5weQ==) | `95.59% <100%> (ø)` | :arrow_up: |
| [pandas/core/indexes/datetimes.py](https://codecov.io/gh/pandas-dev/pandas/pull/24521/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9kYXRldGltZXMucHk=) | `96.27% <100%> (ø)` | :arrow_up: |
| [pandas/core/indexes/api.py](https://codecov.io/gh/pandas-dev/pandas/pull/24521/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9hcGkucHk=) | `99% <100%> (ø)` | :arrow_up: |
| [pandas/core/indexes/interval.py](https://codecov.io/gh/pandas-dev/pandas/pull/24521/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9pbnRlcnZhbC5weQ==) | `95.25% <100%> (-0.02%)` | :arrow_down: |
| [pandas/io/pytables.py](https://codecov.io/gh/pandas-dev/pandas/pull/24521/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3B5dGFibGVzLnB5) | `92.31% <100%> (ø)` | :arrow_up: |
| [pandas/core/indexes/range.py](https://codecov.io/gh/pandas-dev/pandas/pull/24521/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9yYW5nZS5weQ==) | `97.36% <100%> (+0.01%)` | :arrow_up: |
| [pandas/core/indexes/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/24521/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9iYXNlLnB5) | `96.32% <90.9%> (+0.01%)` | :arrow_up: |
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/pull/24521/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `88.04% <0%> (-0.1%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24521?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24521?src=pr&el=footer). Last update [aa05e54...7b15248](https://codecov.io/gh/pandas-dev/pandas/pull/24521?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

so we can followup up on the 2 cases you mentioned at the top of the PR. pls roll them into an issue (you can expand with check boxes if more clear).
Updated the original issue (#24471) with some of the corner cases I mention above. I will try to address these in subsequent PRs once this PR is finished.
Hello @reidy-p! Thanks for updating the PR.



























Cheers ! There are no PEP8 issues in this Pull Request. :beers: 

##### Comment last updated on January 19, 2019 at 18:22 Hours UTC
thanks @reidy-p 
@jreback thanks. I hope to do a few follow-up PRs on this.
Was it intended to change the default behavior of `Index.intersection` here? Previously intersection never sorted, but now it sorts by default.
all of the set ops always sorted; the parameter now allows u to turn it off (default should be the same as before)
This is definitely changed behavior from 0.23.x for intersection (it turned up in a failure in xarray's test suite), but I'll raise another issue about that.
See https://github.com/pandas-dev/pandas/issues/24959 for discussion about the behavior change.