Fix pd.merge to preserve ExtensionArrays dtypes
State: closed

Closes https://github.com/pandas-dev/pandas/issues/20743

Labels:
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
Note to self: 1 actual failing test (rest are network failures):

```
___________________________ TestReshaping.test_merge ___________________________
[gw1] linux -- Python 3.6.5 /home/travis/miniconda3/envs/pandas/bin/python
self = <pandas.tests.extension.json.test_json.TestReshaping object at 0x7fb92d208dd8>
data = JSONArary([{'D': 91, 'k': 84, 'L': 34, 'm': 63, 'C': 8, 'O': 1, 'R': 5}, {'T': 49, 'o': 46}, {'g': 81, 'O': 8, 'p': 54...': 32, 'p': 0, 'J': 89, 'd': 32, 's': 22}, {'W': 70, 'Y': 68, 'D': 87, 'k': 52}, {'F': 29, 'h': 10, 'S': 41, 'y': 24}])
na_value = {}
    def test_merge(self, data, na_value):
    
        df1 = pd.DataFrame({'int1': [1, 2, 3], 'key': [0, 1, 2],
                            'ext': data[:3]})
        df2 = pd.DataFrame({'int2': [1, 2, 3, 4], 'key': [0, 0, 1, 3]})
    
        res = pd.merge(df1, df2)
        exp = pd.DataFrame(
            {'int1': [1, 1, 2], 'int2': [1, 2, 3], 'key': [0, 0, 1],
             'ext': data._constructor_from_sequence(
                 [data[0], data[0], data[1]])})
>       self.assert_frame_equal(res, exp[['ext', 'int1', 'key', 'int2']])
pandas/tests/extension/base/reshaping.py:110: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

E       AssertionError: DataFrame.columns are different
E       
E       DataFrame.columns values are different (75.0 %)
E       [left]:  Index(['int1', 'key', 'ext', 'int2'], dtype='object')
E       [right]: Index(['ext', 'int1', 'key', 'int2'], dtype='object')
```

Not seeing this locally, and also not failing for Decimal, so the column order seems to depend on the extension dtype? (which is a bit strange)
I can reproduce your failure locally. Looking into it now.
Ah, bug in `BaseDecimal.assert_frame_equal` not checking order of all the columns. It only considered extension & non extension columns independently.

So the test is broken for all types, not just JSON.
Pushed a commit fixing the Decimal tests base class and adjust the expected order. Hopefully this will pass things.
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/20745?src=pr&el=h1) Report
> Merging [#20745](https://codecov.io/gh/pandas-dev/pandas/pull/20745?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/78fee04e95e3c53c83c938285580c39e7761ddc8?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/20745/graphs/tree.svg?src=pr&width=650&token=eZ4WkYLtcO&height=150)](https://codecov.io/gh/pandas-dev/pandas/pull/20745?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #20745      +/-   ##
==========================================
+ Coverage   91.84%   91.85%   +<.01%     
==========================================
  Files         153      153              
  Lines       49303    49307       +4     
==========================================
+ Hits        45282    45289       +7     
+ Misses       4021     4018       -3
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.24% <100%> (ø)` | :arrow_up: |
| #single | `41.89% <66.66%> (-0.01%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/20745?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/internals.py](https://codecov.io/gh/pandas-dev/pandas/pull/20745/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW50ZXJuYWxzLnB5) | `95.57% <100%> (+0.03%)` | :arrow_up: |
| [pandas/core/dtypes/common.py](https://codecov.io/gh/pandas-dev/pandas/pull/20745/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZHR5cGVzL2NvbW1vbi5weQ==) | `94.64% <100%> (ø)` | :arrow_up: |
| [pandas/core/indexes/datetimelike.py](https://codecov.io/gh/pandas-dev/pandas/pull/20745/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9kYXRldGltZWxpa2UucHk=) | `96.72% <0%> (ø)` | :arrow_up: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/20745/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.16% <0%> (ø)` | :arrow_up: |
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/pull/20745/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `84.79% <0%> (+0.2%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/20745?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/20745?src=pr&el=footer). Last update [78fee04...9cf8cfe](https://codecov.io/gh/pandas-dev/pandas/pull/20745?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

Thanks for the fix! The order was indeed the culprit, but changed it again to be alphabetically in the original test frame, so it behaves the same for python < 3.6


For Travis, the remaining failure is an s3 one, so this seems to be passing now.