Repository: pandas-dev/pandas
Pull request: BUG: fixed merging with empty frame containing an Int64 column (#25183)
- [x] closes #25183
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Problem statement:
DataFrame.merge with empty frame and Int64 column gives object dtype
#### Code Sample, a copy-pastable example if possible

```python
x = pd.DataFrame({
    'date': pd.Series([pd.Timestamp('2018-01-01')]),
    'value': pd.Series([1], dtype='Int64')
})
y = x[:0]
z = x.merge(y, on='date', how='left')

# z
        date  value_x value_y
0 2018-01-01        1     NaN

# z.dtypes

date       datetime64[ns]
value_x             Int64
value_y            object
dtype: object
```
#### Problem descriptioy

The dtype of the `value_y` column should be `Int64`. 

#### Expected Output

```python
date       datetime64[ns]
value_x             Int64
value_y             Int64
dtype: object
```

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit: None
python: 3.6.8.final.0
python-bits: 64
OS: Linux
OS-release: 4.15.0-43-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: C.UTF-8
LANG: C.UTF-8
LOCALE: en_US.UTF-8
pandas: 0.24.1
pytest: 4.1.1
pip: 18.1
setuptools: 40.6.3
Cython: 0.29.2
numpy: 1.15.4
scipy: None
pyarrow: None
xarray: None
IPython: 7.2.0
sphinx: 1.8.2
patsy: None
dateutil: 2.7.5
pytz: 2018.9
blosc: None
bottleneck: None
tables: 3.4.4
numexpr: 2.6.9
feather: None
matplotlib: 3.0.2
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml.etree: 4.3.0
bs4: None
html5lib: None
sqlalchemy: 1.2.16
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
gcsfs: None

</details>

Hints from issue discussion:
@josham : Given that you have a `NaN` in `value_y`, it would be impossible to have `Int64` (would be a floating dtype actually).  That being said, uncertain as to why you're getting `object` though...

cc @jreback 
I agree that this should be able to return Int types here; this was just put in place in 0.24 so may not be well handled for empty types
@jreback I came across another merge issue with Int64. I can add to this issue or create another, let me know what you prefer.

```python
x = pd.DataFrame({
    'date': [pd.Timestamp('2018-01-01')],
    'value': pd.Series([np.nan], dtype='Int64')
})
y = pd.DataFrame({
    'date': [pd.Timestamp('2018-01-01')],
    'value': pd.Series([4], dtype='Int64')
})
z = x.merge(y, on=['date'])
```
```python
Traceback (most recent call last):
  File "scratch.py", line 23, in <module>
    z = x.merge(y, on=['date'])
  File "/python/lib/python3.6/site-packages/pandas/core/frame.py", line 6868, in merge
    copy=copy, indicator=indicator, validate=validate)
  File "/python/lib/python3.6/site-packages/pandas/core/reshape/merge.py", line 48, in merge
    return op.get_result()
  File "/python/lib/python3.6/site-packages/pandas/core/reshape/merge.py", line 560, in get_result
    concat_axis=0, copy=self.copy)
  File "/python/lib/python3.6/site-packages/pandas/core/internals/managers.py", line 2061, in concatenate_block_managers
    concatenate_join_units(join_units, concat_axis, copy=copy),
  File "/python/lib/python3.6/site-packages/pandas/core/internals/concat.py", line 240, in concatenate_join_units
    for ju in join_units]
  File "/python/lib/python3.6/site-packages/pandas/core/internals/concat.py", line 240, in <listcomp>
    for ju in join_units]
  File "/python/lib/python3.6/site-packages/pandas/core/internals/concat.py", line 194, in get_reindexed_values
    missing_arr = np.empty(self.shape, dtype=empty_dtype)
TypeError: data type not understood
```
@jreback, @TomAugspurger the following seems to fix both issues. Does this seem correct to you?

```
--- a/pandas/core/internals/concat.py
+++ b/pandas/core/internals/concat.py
@@ -190,6 +190,8 @@ class JoinUnit(object):
                     pass
                 elif getattr(self.block, 'is_sparse', False):
                     pass
+                elif getattr(self.block, 'is_extension', False):
+                    pass
                 else:
                     missing_arr = np.empty(self.shape, dtype=empty_dtype)
                     missing_arr.fill(fill_value)
```

Commit message: BUG: fixed merging with empty frame containing an Int64 column (#25183)

Patch to review:
diff --git a/doc/source/whatsnew/v0.24.2.rst b/doc/source/whatsnew/v0.24.2.rst
index f17c4974cd450..aba661bd1fd18 100644
--- a/doc/source/whatsnew/v0.24.2.rst
+++ b/doc/source/whatsnew/v0.24.2.rst
@@ -91,7 +91,7 @@ Bug Fixes
 **Other**
 
 - Bug in :meth:`Series.is_unique` where single occurrences of ``NaN`` were not considered unique (:issue:`25180`)
--
+- Bug in :func:`merge` when merging an empty ``DataFrame`` with an ``Int64`` column or a non-empty ``DataFrame`` with an ``Int64`` column that is all ``NaN`` (:issue:`25183`)
 -
 
 .. _whatsnew_0.242.contributors:
diff --git a/pandas/core/internals/concat.py b/pandas/core/internals/concat.py
index 640587b7f9f31..cb98274962656 100644
--- a/pandas/core/internals/concat.py
+++ b/pandas/core/internals/concat.py
@@ -190,6 +190,8 @@ def get_reindexed_values(self, empty_dtype, upcasted_na):
                     pass
                 elif getattr(self.block, 'is_sparse', False):
                     pass
+                elif getattr(self.block, 'is_extension', False):
+                    pass
                 else:
                     missing_arr = np.empty(self.shape, dtype=empty_dtype)
                     missing_arr.fill(fill_value)
diff --git a/pandas/tests/reshape/merge/test_merge.py b/pandas/tests/reshape/merge/test_merge.py
index 25487ccc76e62..1216730f1862f 100644
--- a/pandas/tests/reshape/merge/test_merge.py
+++ b/pandas/tests/reshape/merge/test_merge.py
@@ -2,6 +2,7 @@
 
 from collections import OrderedDict
 from datetime import date, datetime
+import itertools
 import random
 import re
 
@@ -428,6 +429,64 @@ def check2(exp, kwarg):
                 check1(exp_in, kwarg)
                 check2(exp_out, kwarg)
 
+    @pytest.mark.parametrize(
+        'join_col, val_col', list(itertools.product([
+            pd.Series([1], dtype='int64'),
+            pd.Series([1], dtype='Int64'),
+            pd.Series([1.23]),
+            pd.Series(['foo']),
+            pd.Series([True]),
+            pd.Series([pd.Timestamp('2018-01-01')]),
+            pd.Series([pd.Timestamp('2018-01-01', tz='US/Eastern')]),
+        ], repeat=2)),
+        ids=lambda x: x.dtype.name
+    )
+    def test_merge_empty_frame(self, join_col, val_col):
+        # GH 25183
+        df = pd.DataFrame({'a': join_col, 'b': val_col}, columns=['a', 'b'])
+        df_empty = df[:0]
+        exp = pd.DataFrame({
+            'b_x': pd.Series(dtype=df.dtypes['b']),
+            'a': pd.Series(dtype=df.dtypes['a']),
+            'b_y': pd.Series(dtype=df.dtypes['b']),
+        }, columns=['b_x', 'a', 'b_y'])
+        act = df_empty.merge(df, on='a')
+        assert_frame_equal(act, exp)
+
+    @pytest.mark.parametrize(
+        'join_col, val_col', list(itertools.product(
+            [
+                pd.Series([1], dtype='int64'),
+                pd.Series([1], dtype='Int64'),
+                pd.Series([1.23]),
+                pd.Series(['foo']),
+                pd.Series([True]),
+                pd.Series([pd.Timestamp('2018-01-01')]),
+                pd.Series([pd.Timestamp('2018-01-01', tz='US/Eastern')]),
+            ],
+            [
+                pd.Series([np.nan], dtype='Int64'),
+                pd.Series([np.nan], dtype='float'),
+                pd.Series([np.nan], dtype='object'),
+                pd.Series([pd.NaT]),
+            ]
+        )),
+        ids=lambda x: x.dtype.name
+    )
+    def test_merge_all_na_column(self, join_col, val_col):
+        # GH 25183
+        df_left = pd.DataFrame(
+            {'a': join_col, 'b': val_col}, columns=['a', 'b'])
+        df_right = pd.DataFrame(
+            {'a': join_col, 'b': val_col}, columns=['a', 'b'])
+        exp = pd.DataFrame({
+            'a': join_col,
+            'b_x': val_col,
+            'b_y': val_col,
+        }, columns=['a', 'b_x', 'b_y'])
+        act = df_left.merge(df_right, on='a')
+        assert_frame_equal(act, exp)
+
     def test_merge_nosort(self):
         # #2098, anything to do?
 


Code review comments:
diff_hunk: @@ -428,6 +429,64 @@ def check2(exp, kwarg):
                 check1(exp_in, kwarg)
                 check2(exp_out, kwarg)
 
+    @pytest.mark.parametrize(
+        'join_col, val_col', list(itertools.product([
+            pd.Series([1], dtype='int64'),
+            pd.Series([1], dtype='Int64'),
+            pd.Series([1.23]),
+            pd.Series(['foo']),
+            pd.Series([True]),
+            pd.Series([pd.Timestamp('2018-01-01')]),
+            pd.Series([pd.Timestamp('2018-01-01', tz='US/Eastern')]),
+        ], repeat=2)),
+        ids=lambda x: x.dtype.name
+    )
+    def test_merge_empty_frame(self, join_col, val_col):
+        # GH 25183
+        df = pd.DataFrame({'a': join_col, 'b': val_col}, columns=['a', 'b'])
+        df_empty = df[:0]
+        exp = pd.DataFrame({
+            'b_x': pd.Series(dtype=df.dtypes['b']),
+            'a': pd.Series(dtype=df.dtypes['a']),
+            'b_y': pd.Series(dtype=df.dtypes['b']),
+        }, columns=['b_x', 'a', 'b_y'])
+        act = df_empty.merge(df, on='a')
+        assert_frame_equal(act, exp)
+
+    @pytest.mark.parametrize(
+        'join_col, val_col', list(itertools.product(
path: pandas/tests/reshape/merge/test_merge.py
text: @user1:
you can make these both fixures, they will automatically be the product if you pass both

@author:
done