Repository: pandas-dev/pandas
Pull request: BUG: Fixed concat with reindex and extension types
Closes https://github.com/pandas-dev/pandas/issues/27692
Closes https://github.com/pandas-dev/pandas/issues/33027

I have a larger cleanup planned, but this fixes the linked issues for now.

Problem statement:
BUG: Data types are not preserved while concatenating DataFrames with nullable integers
#### Copy-pastable example

``` python
# input data
import pandas as pd
t1 = pd.DataFrame(index=[0], data={'x':[1]}, dtype='UInt8')
t2 = pd.DataFrame(index=[1], data={'y':[1]}, dtype='UInt8')
t3 = pd.concat([t1,t2], join='outer', sort=False)

'''actual result'''
print(t3.dtypes)
# x    object
# y    object
# dtype: object
```

#### Problem description

Data types are not preserved data type while concatenating DataFrames with nullable integers. Instead, the result of concatenation has mixed data types and so the column types are `object`:

``` python
>>> type(t3.at[0,'x'])
<class 'int'>
>>> type(t3.at[1,'x'])
<class 'float'>
```

#### Expected Output

``` python
'''expected result'''
print(t3.dtypes)
# x    UInt8
# y    UInt8
# dtype: object
```

#### Output of ``pd.show_versions()``

<details>
INSTALLED VERSIONS
------------------
commit           : None
python           : 3.6.3.final.0
python-bits      : 64
OS               : Linux
OS-release       : 3.10.0-862.11.6.el7.x86_64
machine          : x86_64
processor        : x86_64
byteorder        : little
LC_ALL           : en_US.utf-8
LANG             : en_US.utf-8
LOCALE           : en_US.UTF-8

pandas           : 0.25.0
numpy            : 1.16.3
pytz             : 2018.4
dateutil         : 2.7.3
pip              : 19.1.1
setuptools       : 39.0.1
Cython           : 0.29.12
pytest           : 3.3.2
hypothesis       : None
sphinx           : 1.6.6
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : 4.3.4
html5lib         : 0.9999999
pymysql          : None
psycopg2         : None
jinja2           : 2.10
IPython          : 6.2.1
pandas_datareader: None
bs4              : None
bottleneck       : None
fastparquet      : None
gcsfs            : None
lxml.etree       : 4.3.4
matplotlib       : 3.0.2
numexpr          : 2.6.4
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : 0.11.1
pytables         : None
s3fs             : None
scipy            : 1.1.0
sqlalchemy       : None
tables           : 3.5.2
xarray           : None
xlrd             : None
xlwt             : None
xlsxwriter       : None
</details>


concat with Misaligned Column Labels and Extension Types loses Dtype
Was a little surprised by this behavior when combining frames with the string dtype and misaligned labels

```python
In [2]: one = pd.DataFrame([["a"]], dtype="string", columns=["a"])
In [3]: two = pd.DataFrame([["b"]], dtype="string", columns=["b"])
In [4]: pd.concat([one, two]).dtypes
Out[4]:
a    object
b    object
dtype: object
```

A similar issue is exhibited with the integer types:

```python
In [13]: one = pd.DataFrame([[1]], dtype="uint16", columns=["a"])
In [14]: two = pd.DataFrame([[2]], dtype="uint16", columns=["b"])
In [15]: pd.concat([one, two]).dtypes
Out[15]:
a    float64
b    float64
dtype: object
```

Occurs on master

Hints from issue discussion:
Tangentially related to https://github.com/pandas-dev/pandas/issues/22994, which proposes a solution for getting the right dtypes in this situation.

Commit message: BUG: Fixed concat with reindex and extension types

Closes https://github.com/pandas-dev/pandas/issues/27692
Closes https://github.com/pandas-dev/pandas/issues/33027

Patch to review:
diff --git a/doc/source/whatsnew/v1.1.0.rst b/doc/source/whatsnew/v1.1.0.rst
index 2f4e961ff433f..96126c1133a2a 100644
--- a/doc/source/whatsnew/v1.1.0.rst
+++ b/doc/source/whatsnew/v1.1.0.rst
@@ -583,7 +583,7 @@ ExtensionArray
 ^^^^^^^^^^^^^^
 
 - Fixed bug where :meth:`Serires.value_counts` would raise on empty input of ``Int64`` dtype (:issue:`33317`)
--
+- Fixed bug in :func:`concat` when concatenating DataFrames with non-overlaping columns resulting in object-dtype columns rather than preserving the extension dtype (:issue:`27692`, :issue:`33027`)
 
 
 Other
diff --git a/pandas/core/dtypes/concat.py b/pandas/core/dtypes/concat.py
index 301c9bb7b3f5c..b21ed51816edc 100644
--- a/pandas/core/dtypes/concat.py
+++ b/pandas/core/dtypes/concat.py
@@ -68,7 +68,7 @@ def get_dtype_kinds(l):
     return typs
 
 
-def concat_compat(to_concat, axis: int = 0):
+def concat_compat(to_concat, axis: int = 0, ignore_2d_ea: bool = False):
     """
     provide concatenation of an array of arrays each of which is a single
     'normalized' dtypes (in that for example, if it's object, then it is a
@@ -122,7 +122,11 @@ def is_nonempty(x) -> bool:
     any_ea = any(is_extension_array_dtype(x.dtype) for x in to_concat)
 
     if any_ea and axis == 1:
-        to_concat = [np.atleast_2d(x.astype("object")) for x in to_concat]
+        if single_dtype and ignore_2d_ea:
+            cls = type(to_concat[0])
+            return cls._concat_same_type(to_concat)
+        else:
+            to_concat = [np.atleast_2d(x.astype("object")) for x in to_concat]
 
     elif any_ea and single_dtype and axis == 0:
         cls = type(to_concat[0])
diff --git a/pandas/core/internals/concat.py b/pandas/core/internals/concat.py
index 720e6799a3bf3..40671bf9c7638 100644
--- a/pandas/core/internals/concat.py
+++ b/pandas/core/internals/concat.py
@@ -28,7 +28,7 @@
 
 
 def concatenate_block_managers(
-    mgrs_indexers, axes, concat_axis: int, copy: bool
+    mgrs_indexers, axes, concat_axis: int, copy: bool, ignore_2d_ea: bool = False,
 ) -> BlockManager:
     """
     Concatenate block managers into one.
@@ -65,7 +65,9 @@ def concatenate_block_managers(
             b.mgr_locs = placement
         else:
             b = make_block(
-                _concatenate_join_units(join_units, concat_axis, copy=copy),
+                _concatenate_join_units(
+                    join_units, concat_axis, copy=copy, ignore_2d_ea=ignore_2d_ea
+                ),
                 placement=placement,
             )
         blocks.append(b)
@@ -247,6 +249,16 @@ def get_reindexed_values(self, empty_dtype, upcasted_na):
                     pass
                 elif getattr(self.block, "is_extension", False):
                     pass
+                elif is_extension_array_dtype(empty_dtype):
+                    missing_arr = empty_dtype.construct_array_type()._from_sequence(
+                        [], dtype=empty_dtype
+                    )
+                    ncols, nrows = self.shape
+                    assert ncols == 1, ncols
+                    empty_arr = -1 * np.ones((nrows,), dtype="int8")
+                    return missing_arr.take(
+                        empty_arr, allow_fill=True, fill_value=fill_value
+                    )
                 else:
                     missing_arr = np.empty(self.shape, dtype=empty_dtype)
                     missing_arr.fill(fill_value)
@@ -280,7 +292,7 @@ def get_reindexed_values(self, empty_dtype, upcasted_na):
         return values
 
 
-def _concatenate_join_units(join_units, concat_axis, copy):
+def _concatenate_join_units(join_units, concat_axis, copy, ignore_2d_ea=False):
     """
     Concatenate values from several join units along selected axis.
     """
@@ -307,7 +319,9 @@ def _concatenate_join_units(join_units, concat_axis, copy):
             else:
                 concat_values = concat_values.copy()
     else:
-        concat_values = concat_compat(to_concat, axis=concat_axis)
+        concat_values = concat_compat(
+            to_concat, axis=concat_axis, ignore_2d_ea=ignore_2d_ea
+        )
 
     return concat_values
 
@@ -344,6 +358,7 @@ def _get_empty_dtype_and_na(join_units):
 
     upcast_classes = defaultdict(list)
     null_upcast_classes = defaultdict(list)
+
     for dtype, unit in zip(dtypes, join_units):
         if dtype is None:
             continue
@@ -352,6 +367,11 @@ def _get_empty_dtype_and_na(join_units):
             upcast_cls = "category"
         elif is_datetime64tz_dtype(dtype):
             upcast_cls = "datetimetz"
+
+        # may need to move sparse back up
+        elif is_extension_array_dtype(dtype):
+            upcast_cls = "extension"
+
         elif issubclass(dtype.type, np.bool_):
             upcast_cls = "bool"
         elif issubclass(dtype.type, np.object_):
@@ -362,8 +382,6 @@ def _get_empty_dtype_and_na(join_units):
             upcast_cls = "timedelta"
         elif is_sparse(dtype):
             upcast_cls = dtype.subtype.name
-        elif is_extension_array_dtype(dtype):
-            upcast_cls = "object"
         elif is_float_dtype(dtype) or is_numeric_dtype(dtype):
             upcast_cls = dtype.name
         else:
@@ -379,6 +397,12 @@ def _get_empty_dtype_and_na(join_units):
 
     if not upcast_classes:
         upcast_classes = null_upcast_classes
+    if "extension" in upcast_classes:
+        if len(upcast_classes) == 1:
+            cls = upcast_classes["extension"][0]
+            return cls, cls.na_value
+        else:
+            return np.dtype("object"), np.nan
 
     # TODO: de-duplicate with maybe_promote?
     # create the result
diff --git a/pandas/core/reshape/concat.py b/pandas/core/reshape/concat.py
index a868e663b06a5..19ad6d17d4d91 100644
--- a/pandas/core/reshape/concat.py
+++ b/pandas/core/reshape/concat.py
@@ -495,7 +495,11 @@ def get_result(self):
                 mgrs_indexers.append((obj._mgr, indexers))
 
             new_data = concatenate_block_managers(
-                mgrs_indexers, self.new_axes, concat_axis=self.bm_axis, copy=self.copy
+                mgrs_indexers,
+                self.new_axes,
+                concat_axis=self.bm_axis,
+                copy=self.copy,
+                ignore_2d_ea=self.bm_axis == 1 and self._is_frame,
             )
             if not self.copy:
                 new_data._consolidate_inplace()
diff --git a/pandas/tests/extension/base/reshaping.py b/pandas/tests/extension/base/reshaping.py
index c9445ceec2c77..c4628980e1621 100644
--- a/pandas/tests/extension/base/reshaping.py
+++ b/pandas/tests/extension/base/reshaping.py
@@ -107,6 +107,19 @@ def test_concat_extension_arrays_copy_false(self, data, na_value):
         result = pd.concat([df1, df2], axis=1, copy=False)
         self.assert_frame_equal(result, expected)
 
+    def test_concat_with_reindex(self, data):
+        # GH-33027
+        a = pd.DataFrame({"a": data[:5]})
+        b = pd.DataFrame({"b": data[:5]})
+        result = pd.concat([a, b], ignore_index=True)
+        expected = pd.DataFrame(
+            {
+                "a": data.take(list(range(5)) + ([-1] * 5), allow_fill=True),
+                "b": data.take(([-1] * 5) + list(range(5)), allow_fill=True),
+            }
+        )
+        self.assert_frame_equal(result, expected)
+
     def test_align(self, data, na_value):
         a = data[:3]
         b = data[2:5]
diff --git a/pandas/tests/extension/test_categorical.py b/pandas/tests/extension/test_categorical.py
index 059d3453995bd..ed3e55dafab0f 100644
--- a/pandas/tests/extension/test_categorical.py
+++ b/pandas/tests/extension/test_categorical.py
@@ -93,7 +93,8 @@ class TestConstructors(base.BaseConstructorsTests):
 
 
 class TestReshaping(base.BaseReshapingTests):
-    pass
+    def test_concat_with_reindex(self, data):
+        pytest.xfail(reason="Deliberate?")
 
 
 class TestGetitem(base.BaseGetitemTests):


Code review comments:
diff_hunk: @@ -495,7 +495,11 @@ def get_result(self):
                 mgrs_indexers.append((obj._mgr, indexers))
 
             new_data = concatenate_block_managers(
-                mgrs_indexers, self.new_axes, concat_axis=self.bm_axis, copy=self.copy
+                mgrs_indexers,
+                self.new_axes,
+                concat_axis=self.bm_axis,
+                copy=self.copy,
+                ignore_2d_ea=self.bm_axis == 1 and self._is_frame,
path: pandas/core/reshape/concat.py
text: @user1:
this is a really strange keyword name

diff_hunk: @@ -68,7 +68,7 @@ def get_dtype_kinds(l):
     return typs
 
 
-def concat_compat(to_concat, axis: int = 0):
+def concat_compat(to_concat, axis: int = 0, ignore_2d_ea: bool = False):
path: pandas/core/dtypes/concat.py
text: @author:
Will add a doctoring for this later, but the original caller is the only one who knows both the block shape and the extension array shape. So we need to pass that information all the way down to `concat_compat` for whether it's safe to ignore the shape / dim mismatch.

diff_hunk: @@ -495,7 +495,11 @@ def get_result(self):
                 mgrs_indexers.append((obj._mgr, indexers))
 
             new_data = concatenate_block_managers(
-                mgrs_indexers, self.new_axes, concat_axis=self.bm_axis, copy=self.copy
+                mgrs_indexers,
+                self.new_axes,
+                concat_axis=self.bm_axis,
+                copy=self.copy,
+                ignore_2d_ea=self.bm_axis == 1 and self._is_frame,
path: pandas/core/reshape/concat.py
text: @user1:
does anything break if we always ignore_2d_ea?

@user1:
eg i think #33530 is doing something similar to ignore_2d_ea=True

diff_hunk: @@ -122,7 +122,11 @@ def is_nonempty(x) -> bool:
     any_ea = any(is_extension_array_dtype(x.dtype) for x in to_concat)
 
     if any_ea and axis == 1:
-        to_concat = [np.atleast_2d(x.astype("object")) for x in to_concat]
+        if single_dtype and ignore_2d_ea:
path: pandas/core/dtypes/concat.py
text: @user1:
can you mark this with a "# TODO(EA2D): ..."

@author:
I plan to look into the motivation for the atleast_2d first. It seems strange to me that `concat_compat` would be concerned with reshaping, rather than just concatenation. I would think that the caller would take care of that.