Repository: pandas-dev/pandas
Pull request:  BUG-22796 Concat multicolumn tz-aware DataFrame
- [X] closes #22796 
- [X] tests added / passed
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [X] whatsnew entry

Numpy arrays don't have the datetimetz dtype, so I just passed through the DatetimeIndex directly.

Side note: There's another small bug (I think) where np.nan or pd.NaT takes on the dtype of the column instead of the row when concatenating, but the column should instead have an object dtype.


Problem statement:
Concat of dataframe with tz-aware datetime column against dataframe without, fails
```python
a = pd.DataFrame([[1, 2]], dtype='datetime64[ns, UTC]')
b = pd.DataFrame([[3]], dtype='datetime64[ns, UTC]')
pd.concat([a, b])
```

Fails:
```
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-42-457226d62f27> in <module>()
      1 a = pd.DataFrame([[1, 2]], dtype='datetime64[ns, UTC]')
      2 b = pd.DataFrame([[3]], dtype='datetime64[ns, UTC]')
----> 3 pd.concat([a, b])

~/.pyenv/versions/3.6.2/envs/general/lib/python3.6/site-packages/pandas/core/reshape/concat.py in concat(objs, axis, join, join_axes, ignore_index, keys, levels, names, verify_integrity, sort, copy)
    224                        verify_integrity=verify_integrity,
    225                        copy=copy, sort=sort)
--> 226     return op.get_result()
    227 
    228 

~/.pyenv/versions/3.6.2/envs/general/lib/python3.6/site-packages/pandas/core/reshape/concat.py in get_result(self)
    421             new_data = concatenate_block_managers(
    422                 mgrs_indexers, self.new_axes, concat_axis=self.axis,
--> 423                 copy=self.copy)
    424             if not self.copy:
    425                 new_data._consolidate_inplace()

~/.pyenv/versions/3.6.2/envs/general/lib/python3.6/site-packages/pandas/core/internals.py in concatenate_block_managers(mgrs_indexers, axes, concat_axis, copy)
   5419         else:
   5420             b = make_block(
-> 5421                 concatenate_join_units(join_units, concat_axis, copy=copy),
   5422                 placement=placement)
   5423         blocks.append(b)

~/.pyenv/versions/3.6.2/envs/general/lib/python3.6/site-packages/pandas/core/internals.py in concatenate_join_units(join_units, concat_axis, copy)
   5563     to_concat = [ju.get_reindexed_values(empty_dtype=empty_dtype,
   5564                                          upcasted_na=upcasted_na)
-> 5565                  for ju in join_units]
   5566 
   5567     if len(to_concat) == 1:

~/.pyenv/versions/3.6.2/envs/general/lib/python3.6/site-packages/pandas/core/internals.py in <listcomp>(.0)
   5563     to_concat = [ju.get_reindexed_values(empty_dtype=empty_dtype,
   5564                                          upcasted_na=upcasted_na)
-> 5565                  for ju in join_units]
   5566 
   5567     if len(to_concat) == 1:

~/.pyenv/versions/3.6.2/envs/general/lib/python3.6/site-packages/pandas/core/internals.py in get_reindexed_values(self, empty_dtype, upcasted_na)
   5849 
   5850             if not self.indexers:
-> 5851                 if not self.block._can_consolidate:
   5852                     # preserve these for validation in _concat_compat
   5853                     return self.block.values

AttributeError: 'NoneType' object has no attribute '_can_consolidate'
```

#### Output of ``pd.show_versions()``

<details>

[paste the output of ``pd.show_versions()`` here below this line]
INSTALLED VERSIONS
------------------
commit: None
python: 3.6.2.final.0
python-bits: 64
OS: Linux
OS-release: 4.15.0-32-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.23.4
pytest: 3.0.6
pip: 9.0.1
setuptools: 39.1.0
Cython: 0.28.3
numpy: 1.15.1
scipy: 1.1.0
pyarrow: 0.9.0
xarray: 0.10.0
IPython: 6.2.1
sphinx: None
patsy: 0.5.0
dateutil: 2.7.3
pytz: 2018.5
blosc: None
bottleneck: 1.2.1
tables: None
numexpr: 2.6.5
feather: None
matplotlib: 2.2.2
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: 0.9999999
sqlalchemy: 1.1.10
pymysql: None
psycopg2: 2.7.1 (dt dec pq3 ext lo64)
jinja2: 2.9.6
s3fs: None
fastparquet: 0.1.5
pandas_gbq: None
pandas_datareader: None
</details>

Hints from issue discussion:
Thanks for the report, this seems to only be an issue with `axis=0` and merging multicolumn `DataFrames` with tz-aware data

I suspect it has something to do with the fact that the DatetimeTZBlock only stores data in a 1D array. Investigations welcome!
```
In [13]: pd.__version__
Out[13]: '0.24.0.dev0+624.g4612a8282'

In [14]: a = pd.DataFrame([[1, 2]], dtype='datetime64[ns, UTC]')
    ...: b = pd.DataFrame([[3]], dtype='datetime64[ns, UTC]')
    ...: pd.concat([a, b], axis=1)
    ...:
    ...:
Out[14]:
                                    0                 ...                                                   0
0 1970-01-01 00:00:00.000000001+00:00                 ...                 1970-01-01 00:00:00.000000003+00:00

[1 rows x 3 columns]

In [25]: a = pd.DataFrame([[1, 2]], dtype='datetime64[ns, UTC]')
    ...: b = pd.DataFrame([[3]], dtype='datetime64[ns, UTC]')
    ...: pd.concat([a.iloc[:, [0]], b])
    ...:
    ...:
Out[25]:
                                    0
0 1970-01-01 00:00:00.000000001+00:00
0 1970-01-01 00:00:00.000000003+00:00
```
@mroeschke @bchu   i think that  it is logical that they didn't allow the option of axis = 0 for this case.
Let say instead of date time we had numeric data like
```import pandas as pd
print(pd.__version__)
a = pd.DataFrame([[1, 2]])
b = pd.DataFrame([[3]] )
c = pd.concat([a, b],axis=0)
print(a)
print(b)
print(c)

the output would be something like
a
   0  1
0  1  2
b
   0
0  3
c
   0    1
0  1  2.0
0  3  NaN


the thing is that when there was not a number python defaulted to NaN
but if we were to do the same thing with datetime there is not an NaD(Not a Date) and the 1970 
00:00:00 would be a wrong value to put there so the behavior is already correct that's why i think that 
this operation with axis 0 should not be permitted. 
The missing value should be filled with ``NaT``

Commit message: use construct_array_type

Patch to review:
diff --git a/doc/source/whatsnew/v0.24.0.txt b/doc/source/whatsnew/v0.24.0.txt
index 56dd6c82681eb..f1a953d3a4258 100644
--- a/doc/source/whatsnew/v0.24.0.txt
+++ b/doc/source/whatsnew/v0.24.0.txt
@@ -892,6 +892,7 @@ Reshaping
 - Bug in :func:`pandas.wide_to_long` when a string is passed to the stubnames argument and a column name is a substring of that stubname (:issue:`22468`)
 - Bug in :func:`merge` when merging ``datetime64[ns, tz]`` data that contained a DST transition (:issue:`18885`)
 - Bug in :func:`merge_asof` when merging on float values within defined tolerance (:issue:`22981`)
+- Bug in :func:`pandas.concat` when concatenating a multicolumn DataFrame with tz-aware data against a DataFrame with a different number of columns (:issue`22796`)
 
 Build Changes
 ^^^^^^^^^^^^^
diff --git a/pandas/core/dtypes/dtypes.py b/pandas/core/dtypes/dtypes.py
index 611cae28877c3..f07fb3cd80eab 100644
--- a/pandas/core/dtypes/dtypes.py
+++ b/pandas/core/dtypes/dtypes.py
@@ -546,6 +546,17 @@ def __new__(cls, unit=None, tz=None):
             cls._cache[key] = u
             return u
 
+    @classmethod
+    def construct_array_type(cls):
+        """Return the array type associated with this dtype
+
+        Returns
+        -------
+        type
+        """
+        from pandas import DatetimeIndex
+        return DatetimeIndex
+
     @classmethod
     def construct_from_string(cls, string):
         """ attempt to construct this type from a string, raise a TypeError if
diff --git a/pandas/core/internals/concat.py b/pandas/core/internals/concat.py
index 5a3f11525acf8..17b5a8e81a792 100644
--- a/pandas/core/internals/concat.py
+++ b/pandas/core/internals/concat.py
@@ -186,6 +186,10 @@ def get_reindexed_values(self, empty_dtype, upcasted_na):
 
                 if getattr(self.block, 'is_datetimetz', False) or \
                         is_datetimetz(empty_dtype):
+                    if self.block is None:
+                        array = empty_dtype.construct_array_type()
+                        missing_arr = array([fill_value], dtype=empty_dtype)
+                        return missing_arr.repeat(self.shape[0])
                     pass
                 elif getattr(self.block, 'is_categorical', False):
                     pass
diff --git a/pandas/tests/frame/test_combine_concat.py b/pandas/tests/frame/test_combine_concat.py
index 15ca65395e4fc..9cb50b9602eda 100644
--- a/pandas/tests/frame/test_combine_concat.py
+++ b/pandas/tests/frame/test_combine_concat.py
@@ -17,6 +17,8 @@
 import pandas.util.testing as tm
 from pandas.util.testing import assert_frame_equal, assert_series_equal
 
+import pytest
+
 
 class TestDataFrameConcatCommon(TestData):
 
@@ -53,6 +55,27 @@ def test_concat_multiple_tzs(self):
         expected = DataFrame(dict(time=[ts2, ts3]))
         assert_frame_equal(results, expected)
 
+    @pytest.mark.parametrize(
+        't1',
+        [
+            '2015-01-01',
+            pytest.param(pd.NaT, marks=pytest.mark.xfail(
+                reason='GH23037 incorrect dtype when concatenating'))])
+    def test_concat_tz_NaT(self, t1):
+        # GH 22796
+        # Concating tz-aware multicolumn DataFrames
+        ts1 = Timestamp(t1, tz='UTC')
+        ts2 = Timestamp('2015-01-01', tz='UTC')
+        ts3 = Timestamp('2015-01-01', tz='UTC')
+
+        df1 = DataFrame([[ts1, ts2]])
+        df2 = DataFrame([[ts3]])
+
+        result = pd.concat([df1, df2])
+        expected = DataFrame([[ts1, ts2], [ts3, pd.NaT]], index=[0, 0])
+
+        assert_frame_equal(result, expected)
+
     def test_concat_tuple_keys(self):
         # GH 14438
         df1 = pd.DataFrame(np.ones((2, 2)), columns=list('AB'))


Code review comments:
diff_hunk: @@ -55,9 +55,12 @@ def test_concat_multiple_tzs(self):
         expected = DataFrame(dict(time=[ts2, ts3]))
         assert_frame_equal(results, expected)
 
-    @pytest.mark.parametrize('t1', ['2015-01-01',
-        pytest.param(pd.NaT, marks=pytest.mark.xfail(
-                     reason='GH23037 incorrect dtype when concatenating'))])
+    @pytest.mark.parametrize(
+        't1',
+        [
+            '2015-01-01',
+            pytest.param(pd.NaT, marks=pytest.mark.xfail(
+                reason='GH23037 incorrect dtype when concatenating'))])
path: pandas/tests/frame/test_combine_concat.py
text: @user1:
Pass `strict=True` to the xfail so that we remember to unxfail it?

@user2:
good point.