BUG: Fix concat of frames with extension types (no reindexed columns)
State: closed

This PR tries to fix the case of concatting dataframes with extension dtypes and non-extension dtypes, like this example (giving object dtype on master, instead of preserving Int64 dtype):

```
In [15]: df1 = pd.DataFrame({"a": pd.array([1, 2, 3], dtype="Int64")}) 
    ...: df2 = pd.DataFrame({"a": np.array([4, 5, 6])})  

In [16]: pd.concat([df1, df2]) 
Out[16]: 
   a
0  1
1  2
2  3
0  4
1  5
2  6

In [17]: pd.concat([df1, df2]).dtypes 
Out[17]: 
a    object
dtype: object
```

Up to now, the `_get_common_dtype` machinery of the ExtensionArrays was only used for concatting Series (`axis=0` in `concat_compat`). Now I also try to make use of this for DataFrame columns, if there is any extension dtype in the to-concat arrays. This is checked in `_concatenate_join_units`. And we need to handle the dimension-mismach. I decided to handle that inside the internals code (reducing the dimension for non-EAs, and if the concat result is not an EA, adding back a dimension), so `concat_compat` does not need to be aware of this.


Partly overlapping with https://github.com/pandas-dev/pandas/pull/33522 (cc @TomAugspurger), but this PR does *not yet* fix the original case the other PR is fixing: concat with reindexed (thus all missing) columns becoming object instead of preserving object dtype.

Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
Hello @jorisvandenbossche! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:













There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-06-18 13:27:00 UTC
cc @TomAugspurger @jbrockmendel 
> I didn't immediately see the reason for your changesin dtypes/concat, but I suppose it's related to

So the change in `dtypes/concat.py` is `if any_ea and axis == 0:` changed to `if any_ea:`, ensuring that we now take the EA-specific code path for both axis=0 and axis=1. 

Now, the axis is a bit confusing here, as we are actually always concatting the EAs in the same way (making one long EA), but the axis comes from the "higher" level whether we are concatting series or frames. 
Now, when it comes to concatting EAs, this axis thus doesn't matter. 

So that is indeed as you say related to the fact that I now also take the EA-specific code path for DataFrame columns (which result in axis=1) and not only for Series (axis=0).

When it comes to `concat_compat` and concatting EAs, the `axis` basically can be ignored.


> Do we need a whatsnew note?

I added a general note about concat with extension arrays (that also references the previous PR that made this possible for Series).
This should be ready for review / merge now.
Planning to merge tomorrow if there are no objections.
Sorry @jorisvandenbossche, one more merge? :)
@jorisvandenbossche IIRC there was some discussion of implementing Categorical._concat_same_type directly in the method and having union_categoricals defer to the method rather than the other way around.  Is that awaiting consensus or just awaiting implementation?
@jbrockmendel the discussion was here: https://github.com/pandas-dev/pandas/pull/33607#discussion_r410281807, AFAIK just awaiting someone doing it