DEPR: join_axes-kwarg in pd.concat
State: closed

- [x] closes #21951 
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Deprecate
description: Functionality to remove in pandas

Comments:
Hello @h-vetinari! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:











There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-07-03 02:21:51 UTC
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/22318?src=pr&el=h1) Report
> Merging [#22318](https://codecov.io/gh/pandas-dev/pandas/pull/22318?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/b618eb6ecb19f243f9f68e73f56a67f3f629e09d?src=pr&el=desc) will **increase** coverage by `50.26%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/22318/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/22318?src=pr&el=tree)

```diff
@@             Coverage Diff             @@
##           master   #22318       +/-   ##
===========================================
+ Coverage   41.96%   92.23%   +50.26%     
===========================================
  Files         180      161       -19     
  Lines       50860    51333      +473     
===========================================
+ Hits        21345    47348    +26003     
+ Misses      29515     3985    -25530
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.62% <100%> (?)` | |
| #single | `42.29% <34.48%> (+0.32%)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/22318?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/groupby/generic.py](https://codecov.io/gh/pandas-dev/pandas/pull/22318/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS9nZW5lcmljLnB5) | `87.1% <100%> (+72.32%)` | :arrow_up: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/22318/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.03% <100%> (+62.07%)` | :arrow_up: |
| [pandas/core/reshape/concat.py](https://codecov.io/gh/pandas-dev/pandas/pull/22318/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvcmVzaGFwZS9jb25jYXQucHk=) | `97.66% <100%> (+53.66%)` | :arrow_up: |
| [pandas/core/generic.py](https://codecov.io/gh/pandas-dev/pandas/pull/22318/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ2VuZXJpYy5weQ==) | `96.81% <100%> (+58.18%)` | :arrow_up: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/22318/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :arrow_down: |
| [pandas/compat/\_\_init\_\_.py](https://codecov.io/gh/pandas-dev/pandas/pull/22318/diff?src=pr&el=tree#diff-cGFuZGFzL2NvbXBhdC9fX2luaXRfXy5weQ==) | `58.36% <0%> (-33.64%)` | :arrow_down: |
| [pandas/io/clipboard/clipboards.py](https://codecov.io/gh/pandas-dev/pandas/pull/22318/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2NsaXBib2FyZC9jbGlwYm9hcmRzLnB5) | `28.23% <0%> (-6.55%)` | :arrow_down: |
| [pandas/plotting/\_misc.py](https://codecov.io/gh/pandas-dev/pandas/pull/22318/diff?src=pr&el=tree#diff-cGFuZGFzL3Bsb3R0aW5nL19taXNjLnB5) | `38.98% <0%> (-4.26%)` | :arrow_down: |
| [pandas/io/formats/console.py](https://codecov.io/gh/pandas-dev/pandas/pull/22318/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvY29uc29sZS5weQ==) | `75.75% <0%> (-2.37%)` | :arrow_down: |
| [pandas/\_libs/\_\_init\_\_.py](https://codecov.io/gh/pandas-dev/pandas/pull/22318/diff?src=pr&el=tree#diff-cGFuZGFzL19saWJzL19faW5pdF9fLnB5) | `100% <0%> (ø)` | :arrow_up: |
| ... and [174 more](https://codecov.io/gh/pandas-dev/pandas/pull/22318/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/22318?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/22318?src=pr&el=footer). Last update [b618eb6...154ab0e](https://codecov.io/gh/pandas-dev/pandas/pull/22318?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

@jorisvandenbossche you had some comments on the issue. I am all for making things simpler.
@jorisvandenbossche 
I'm answering your comment from #21951 here, as this is further ahead:

>"but this will be fixed by 21160 anyway." -> that is long off to be the default integer type, so I don't think we should use that as an argument now

Fair enough, that was a misunderstanding on my part. But the dtype-question is a side-issue here, IMO

> "with reindex and reindex_like, it is redundant as well" -> I don't know the implementation, but I would assume that reindexing after the fact can be less performant? (assuming that with `join_axes` it reindexes each input before concatenating)

I posed this question in the OP ("Only question is if performance would be much worse, if concatenating huge Series/DFs before selecting small index-subset."), and it's a valid point. The user *could* apply a reindex directly to the arguments of `concat` (I originally had something to that effect in the `FutureWarning` in this PR), but I see that it would be good to keep having a way to do this easily.

> It is certainly true the way it is explained and spelled (eg the fact that you need to pass a list) is certainly outdated now Panel is removed. But we could also consider improving it.

In my opinion, the `join_axes`-keyword (as a name, not necessarily as functionality) should definitely be deprecated. The question of what to replace it with is very similar to the discussion in https://github.com/pandas-dev/pandas/issues/21855#issuecomment-413103937.

There might be a good solution for replacing the `join_axes`-functionality with something like the following (conceptually):
```
concat(objs, axis=0, join='outer', index=None, columns=None, ignore_index=False,
       keys=None, levels=None, names=None, verify_integrity=False, sort=None, copy=True)
[...]
if (axis == 0 and index is not None) or (axis == 1 and columns is not None):
   raise ValueError('Cannot set index of concatenation axis; '
                    'can only set index of non-concatenation axis')
index = join if index is None else index
columns = join if columns is None else columns
[...]
objs = [x.reindex(index/columns, axis=other_axis) for x in objs]
[...]
# actual concatenation
```
Then index the non-concatenation index of the result could be set as desired (essentially replacing `join_axes` with a cleaner API, now that `Panel` will be deprecated).

> (note that I am not married to the keyword, I have never used it myself, but just think we should have a bit more discussion about it. It would be interesting to search for usage on github/SO)

There's not much on SO - https://www.google.com/search?q=stackoverflow+pandas+concat+join_axes yields only one (sorta) relevant hit: https://stackoverflow.com/q/27391081

Neither did I find much of substance here: https://github.com/search?p=2&q=join_axes&type=Issues
@jreback 
> small comment, lgtm. ping on green.

All green (rebased due to persistent test failures).
@jreback

Green and all feedback incorporated. Anything missing?
@jorisvandenbossche 

I answered above (https://github.com/pandas-dev/pandas/pull/22318#issuecomment-415448620) to your remarks/questions in the issue. Any further comments, API- or otherwise?
@jreback 
> some comments. ping on green.

ping on green
@jorisvandenbossche i am ok with this
any objections
@h-vetinari sorry for my slow follow-up, didn't have much time the last weeks

I also saw that the discussion here (about an alternative way to do it) is related to the joining discussions in other PRs/issues (you linked https://github.com/pandas-dev/pandas/issues/21855). I will try to take a look at those discussions one of the coming days.

I personally think we should first decide on an alternative (or decide upon if we want one) before actually deprecating this, but no strong opinion.

BTW, if we keep the PR as is, the documentation certainly needs an update (there are example of `join_axes` in the tutorial docs).
@jorisvandenbossche 
> I will try to take a look at those discussions one of the coming days.

Any luck with that?

As a potential shortcut to reading that other discussion, you could have a look at the pseudo-implementation I wrote above in https://github.com/pandas-dev/pandas/pull/22318#issuecomment-415448620.

Barring a decision on what to replace the API with, which tutorial docs do you mean? I didn't find anything in https://pandas.pydata.org/pandas-docs/stable/tutorials.html, but there's a bit in `doc/source/merging.rst`.
looks ok to me, @jorisvandenbossche has some questions.
@jreback @jorisvandenbossche 
> Barring a decision on what to replace the API with, which tutorial docs do you mean? I didn't find anything in https://pandas.pydata.org/pandas-docs/stable/tutorials.html, but there's a bit in doc/source/merging.rst.
@jorisvandenbossche 
Can we pick this up again?
@gfyoung 
> @h-vetinari : FYI, instead of pushing commits to trigger entirely new builds, just ping one of us. We can restart individual builds for you.

Could you retrigger that timed-out travis job please? ;-)
revisiting, can you rebase. 
@jreback @jorisvandenbossche 
Rebased and green
@jreback 
> revisiting, can you rebase.

Merged master and green.
@jorisvandenbossche last look. This has been in the queue for a while and is a valid deprecation.
@jreback @jorisvandenbossche 
Last ping (in case you consider this relevant for the RC).
@h-vetinari can you merge master on this?
Comment from above:

> BTW, if we keep the PR as is, the documentation certainly needs an update (there are example of `join_axes` in the tutorial docs).

Did you address that already?

As I said above (https://github.com/pandas-dev/pandas/pull/22318#issuecomment-419139173), I might prefer a conclusion first to the align axes discussion, but since that is also not very active / close to conclusion, I won't hold up this PR if the other core devs are in favor of merging it.
@jorisvandenbossche

I answered that exact point further up the thread:

> @h-vetinari: Barring a decision on what to replace the API with, which tutorial docs do you mean? I didn't find anything in https://pandas.pydata.org/pandas-docs/stable/tutorials.html, but there's a bit in doc/source/merging.rst.
> but there's a bit in doc/source/merging.rst.

Yes, exactly that (sorry, my "tutorial docs" was meaning the written rst docs (like merging.rst) to contrast it to docstrings, which was probably not clear :-))
@jorisvandenbossche 
Changed the part in `merging.rst`, PTAL.
this deprecation looks ok to me. pls merge master.
if @pandas-dev/pandas-core wants to finish this up
@jreback
Couldn't respond to your last ping at the time and then forgot. Merged master.
@jreback

This should be done. Could you please restart the timed-out travis job?
From a discussion with @jreback: it might be useful to add a `join='left'` option for one use case of `join_axes` to use the index of the first dataframe in the list.