Repository: pandas-dev/pandas

Issue:
concat doesn't work with mixed Series/DataFrames
concat thows an `AssertionError` when passed a ciollection of mixed Series and Dataframes e.g.

``` python
s1 = pd.TimeSeries(np.sin(linspace(0, 2*pi, 100)), 
                   index=pd.date_range('01-Jan-2013', 
periods=100, freq='H')) 

s2 = pd.TimeSeries(np.cos(linspace(0, 2*pi, 100)), 
                  index=pd.date_range('01-Jan-2013', 
periods=100, freq='H')) 

df = pd.DataFrame(np.cos(linspace(0, 2*pi, 
100)).reshape(-1,1), 
                   index=pd.date_range('01-Jan-2013', 
periods=100, freq='H')) 

In [23]: pd.concat([df,df], axis=1).shape 
Out[23]: (100, 2) 

In [24]: pd.concat([s1,s2], axis=1).shape 
Out[24]: (100, 2) 

In [25]: pd.concat([s1,s2,s1], axis=1).shape 
Out[25]: (100, 3) 

In [26]: pd.concat([s1,df,s2], axis=1).shape 
Traceback (most recent call last):

  File "<ipython-input-9-4512588e71f2>", line 1, in <module>
    pd.concat([s1,df,s2], axis=1).shape

  File "c:\dev\code\pandas\pandas\tools\merge.py", line 881, in concat
    return op.get_result()

  File "c:\dev\code\pandas\pandas\tools\merge.py", line 960, in get_result
    columns=self.new_axes[1])

  File "c:\dev\code\pandas\pandas\core\frame.py", line 376, in __init__
    mgr = self._init_dict(data, index, columns, dtype=dtype)

  File "c:\dev\code\pandas\pandas\core\frame.py", line 505, in _init_dict
    dtype=dtype)

  File "c:\dev\code\pandas\pandas\core\frame.py", line 5181, in _arrays_to_mgr
    mgr = BlockManager(blocks, axes)

  File "c:\dev\code\pandas\pandas\core\internals.py", line 499, in __init__
    self._verify_integrity()

  File "c:\dev\code\pandas\pandas\core\internals.py", line 584, in _verify_integrity
    raise AssertionError('Block shape incompatible with manager')

AssertionError: Block shape incompatible with manager

In [27]: pd.__version__
Out[27]: '0.10.0.dev-fbd77d5'
```



Discussion:
Well that's not very helpful. Should be improved at some point

Am I right in saying these give better errors in dev?

```
pd.concat([s1,df,s2])
ValueError: arrays must have same number of dimensions

pd.concat([s1,df,s2], axis=1)
ValueError: Shape of passed values is (3,), indices imply (3, 100)
```

I like the first one, prob should catch the error in concat and put up a better message for the 2nd one though....

Agreed.

ok...let's move to 0.12 then...

It's not the incorrect error which is the problem, but the fact that an error is thrown at all. This makes it difficult/tedious to write generic code that works whether or not the input is a DataFrame or a TimeSeries.

In this case the TimeSeries should be treated as a DataFrame with one column where the column label is taken from the TimeSeries name. It the axis parameter was 0 then the TimeSeries should be treated as a DataFrame with one row where the row label is taken from the TimeSeries name.

@dhirschfeld I think we will have a detailed look at this, but nothing is stopping you from pre-conditining, e.g.

```
 pd.concat([ pd.DataFrame(x) for x in [s1,df,s2] ], axis=1).shape
```

The DataFrame constructor does exactly what you want for a series with axis=1, and passed thru a DataFrame
(axis=0 you would need to pass the orient so it transposes I think)

I also stumbled on this, but I saw there is already this issue for it. 

I think in current master it is still very confusing. An example (with the series being a column from a dataframe, full example: http://nbviewer.ipython.org/5868420/bug_pandas-concat-series.ipynb ):
- concat with `[df1, df2[col]]`: `IndexError: list index out of range`
- concat with `[df1[col], df2]`: `AttributeError: 'DataFrame' object has no attribute 'name'`

I think at the moment it is not clear from the docs that this is not allowed (mixing dataframes with series/columns), and when you do it, you don't get a very informative error message.

So should work like this:

```
In [1]: df = pd.DataFrame([[1, 2], [3, 4]])

In [2]: s = pd.Series([5, 6], name=3)

In [3]: pd.concat([df, pd.DataFrame(s).T])  # but with [df, s]
Out[3]:
   0  1
0  1  2
1  3  4
3  5  6

In [4]: pd.concat([df, pd.DataFrame(s)], axis=1)  # but with [df. s]
Out[4]:
   0  1  3
0  1  2  5
1  3  4  6

# also both with [s, df]
```

series must have a name FYI (otherwise needs to raise)

atm it's a bit sketchy with name:

```
In [5]: s1 = pd.Series([5, 6])

In [6]: pd.concat([s, s1], axis=1)   # loses name of s
Out[6]:
   0  1
0  5  5
1  6  6
```

Not sure should always raise w/o name.

@hayd that's the problem, it "works" but really should not (as its assigned as if the column index doesn't exist). I would just be careful about it, maybe only allowing it if `as_index=False` or series has a name

not exactly the same situation, but `append` raises if the appended series doesn't have a name and axis=0, so certainly precedent for it.


Resolving patch:
diff --git a/pandas/core/groupby.py b/pandas/core/groupby.py
--- a/pandas/core/groupby.py
+++ b/pandas/core/groupby.py
@@ -2209,10 +2209,7 @@ def _wrap_applied_output(self, keys, values, not_indexed_same=False):
 
             # make Nones an empty object
             if com._count_not_none(*values) != len(values):
-                v = None
-                for v in values:
-                    if v is not None:
-                        break
+                v = next(v for v in values if v is not None)
                 if v is None:
                     return DataFrame()
                 elif isinstance(v, NDFrame):
diff --git a/pandas/tools/merge.py b/pandas/tools/merge.py
--- a/pandas/tools/merge.py
+++ b/pandas/tools/merge.py
@@ -957,7 +957,7 @@ def __init__(self, objs, axis=0, join='outer', join_axes=None,
             objs = [objs[k] for k in keys]
 
         if keys is None:
-            objs = [obj for obj in objs if obj is not None]
+            objs = [obj for obj in objs if obj is not None ]
         else:
             # #1649
             clean_keys = []
@@ -973,16 +973,43 @@ def __init__(self, objs, axis=0, join='outer', join_axes=None,
         if len(objs) == 0:
             raise Exception('All objects passed were None')
 
-        # consolidate data
+        # consolidate data & figure out what our result ndim is going to be
+        ndims = set()
         for obj in objs:
-            if isinstance(obj, NDFrame):
-                obj.consolidate(inplace=True)
-        self.objs = objs
+            if not isinstance(obj, NDFrame):
+                raise TypeError("cannot concatenate a non-NDFrame object")
+
+            # consolidate
+            obj.consolidate(inplace=True)
+            ndims.add(obj.ndim)
+
+        # get the sample
+        # want the higest ndim that we have, and must be non-empty
+        # unless all objs are empty
+        sample = None
+        if len(ndims) > 1:
+            max_ndim = max(ndims)
+            for obj in objs:
+                if obj.ndim == max_ndim and np.sum(obj.shape):
+                    sample = obj
+                    break
 
-        sample = objs[0]
+        else:
+            # filter out the empties
+            # if we have not multi-index possibiltes
+            df = DataFrame([ obj.shape for obj in objs ]).sum(1)
+            non_empties = df[df!=0]
+            if len(non_empties) and (keys is None and names is None and levels is None and join_axes is None):
+                objs = [ objs[i] for i in non_empties.index ]
+                sample = objs[0]
+
+        if sample is None:
+            sample = objs[0]
+        self.objs = objs
 
         # Need to flip BlockManager axis in the DataFrame special case
-        if isinstance(sample, DataFrame):
+        self._is_frame = isinstance(sample, DataFrame)
+        if self._is_frame:
             axis = 1 if axis == 0 else 0
 
         self._is_series = isinstance(sample, ABCSeries)
@@ -990,11 +1017,39 @@ def __init__(self, objs, axis=0, join='outer', join_axes=None,
             raise AssertionError("axis must be between 0 and {0}, "
                                  "input was {1}".format(sample.ndim, axis))
 
+        # if we have mixed ndims, then convert to highest ndim
+        # creating column numbers as needed
+        if len(ndims) > 1:
+            current_column = 0
+            max_ndim = sample.ndim
+            self.objs, objs = [], self.objs
+            for obj in objs:
+
+                ndim = obj.ndim
+                if ndim == max_ndim:
+                    pass
+
+                elif ndim != max_ndim-1:
+                    raise ValueError("cannot concatenate unaligned mixed "
+                                     "dimensional NDFrame objects")
+
+                else:
+                    name = getattr(obj,'name',None)
+                    if ignore_index or name is None:
+                        name = current_column
+                        current_column += 1
+
+                    # doing a row-wise concatenation so need everything
+                    # to line up
+                    if self._is_frame and axis == 1:
+                        name = 0
+                    obj = sample._constructor({ name : obj })
+
+                self.objs.append(obj)
+
         # note: this is the BlockManager axis (since DataFrame is transposed)
         self.axis = axis
-
         self.join_axes = join_axes
-
         self.keys = keys
         self.names = names
         self.levels = levels
diff --git a/vb_suite/join_merge.py b/vb_suite/join_merge.py
--- a/vb_suite/join_merge.py
+++ b/vb_suite/join_merge.py
@@ -186,6 +186,21 @@ def sample(values, k):
 concat_small_frames = Benchmark('concat([df] * 1000)', setup,
                                 start_date=datetime(2012, 1, 1))
 
+
+#----------------------------------------------------------------------
+# Concat empty
+
+setup = common_setup + """
+df = DataFrame(dict(A = range(10000)),index=date_range('20130101',periods=10000,freq='s'))
+empty = DataFrame()
+"""
+
+concat_empty_frames1 = Benchmark('concat([df,empty])', setup,
+                                start_date=datetime(2012, 1, 1))
+concat_empty_frames2 = Benchmark('concat([empty,df])', setup,
+                                start_date=datetime(2012, 1, 1))
+
+
 #----------------------------------------------------------------------
 # Ordered merge
 


Test patch:
diff --git a/pandas/tools/tests/test_merge.py b/pandas/tools/tests/test_merge.py
--- a/pandas/tools/tests/test_merge.py
+++ b/pandas/tools/tests/test_merge.py
@@ -1653,6 +1653,77 @@ def test_handle_empty_objects(self):
 
         tm.assert_frame_equal(concatted, expected)
 
+        # empty as first element with time series
+        # GH3259
+        df = DataFrame(dict(A = range(10000)),index=date_range('20130101',periods=10000,freq='s'))
+        empty = DataFrame()
+        result = concat([df,empty],axis=1)
+        assert_frame_equal(result, df)
+        result = concat([empty,df],axis=1)
+        assert_frame_equal(result, df)
+
+        result = concat([df,empty])
+        assert_frame_equal(result, df)
+        result = concat([empty,df])
+        assert_frame_equal(result, df)
+
+    def test_concat_mixed_objs(self):
+
+        # concat mixed series/frames
+        # G2385
+
+        # axis 1
+        index=date_range('01-Jan-2013', periods=10, freq='H')
+        arr = np.arange(10, dtype='int64')
+        s1 = Series(arr, index=index)
+        s2 = Series(arr, index=index)
+        df = DataFrame(arr.reshape(-1,1), index=index)
+
+        expected = DataFrame(np.repeat(arr,2).reshape(-1,2), index=index, columns = [0, 0])
+        result = concat([df,df], axis=1)
+        assert_frame_equal(result, expected)
+
+        expected = DataFrame(np.repeat(arr,2).reshape(-1,2), index=index, columns = [0, 1])
+        result = concat([s1,s2], axis=1)
+        assert_frame_equal(result, expected)
+
+        expected = DataFrame(np.repeat(arr,3).reshape(-1,3), index=index, columns = [0, 1, 2])
+        result = concat([s1,s2,s1], axis=1)
+        assert_frame_equal(result, expected)
+
+        expected = DataFrame(np.repeat(arr,5).reshape(-1,5), index=index, columns = [0, 0, 1, 2, 3])
+        result = concat([s1,df,s2,s2,s1], axis=1)
+        assert_frame_equal(result, expected)
+
+        # with names
+        s1.name = 'foo'
+        expected = DataFrame(np.repeat(arr,3).reshape(-1,3), index=index, columns = ['foo', 0, 0])
+        result = concat([s1,df,s2], axis=1)
+        assert_frame_equal(result, expected)
+
+        s2.name = 'bar'
+        expected = DataFrame(np.repeat(arr,3).reshape(-1,3), index=index, columns = ['foo', 0, 'bar'])
+        result = concat([s1,df,s2], axis=1)
+        assert_frame_equal(result, expected)
+
+        # ignore index
+        expected = DataFrame(np.repeat(arr,3).reshape(-1,3), index=index, columns = [0, 1, 2])
+        result = concat([s1,df,s2], axis=1, ignore_index=True)
+        assert_frame_equal(result, expected)
+
+        # axis 0
+        expected = DataFrame(np.tile(arr,3).reshape(-1,1), index=index.tolist() * 3, columns = [0])
+        result = concat([s1,df,s2])
+        assert_frame_equal(result, expected)
+
+        expected = DataFrame(np.tile(arr,3).reshape(-1,1), columns = [0])
+        result = concat([s1,df,s2], ignore_index=True)
+        assert_frame_equal(result, expected)
+
+        # invalid concatente of mixed dims
+        panel = tm.makePanel()
+        self.assertRaises(ValueError, lambda : concat([panel,s1],axis=1))
+
     def test_panel_join(self):
         panel = tm.makePanel()
         tm.add_nans(panel)
@@ -1967,6 +2038,13 @@ def test_concat_series_axis1_same_names_ignore_index(self):
         result = concat([s1, s2], axis=1, ignore_index=True)
         self.assertTrue(np.array_equal(result.columns, [0, 1]))
 
+    def test_concat_invalid(self):
+
+        # trying to concat a ndframe with a non-ndframe
+        df1 = mkdf(10, 2)
+        for obj in [1, dict(), [1, 2], (1, 2) ]:
+            self.assertRaises(TypeError, lambda x: concat([ df1, obj ]))
+
     def test_concat_invalid_first_argument(self):
         df1 = mkdf(10, 2)
         df2 = mkdf(10, 2)
@@ -1975,15 +2053,6 @@ def test_concat_invalid_first_argument(self):
         # generator ok though
         concat(DataFrame(np.random.rand(5,5)) for _ in range(3))
 
-    def test_concat_mixed_types_fails(self):
-        df = DataFrame(randn(10, 1))
-
-        with tm.assertRaisesRegexp(TypeError, "Cannot concatenate.+"):
-            concat([df[0], df], axis=1)
-
-        with tm.assertRaisesRegexp(TypeError, "Cannot concatenate.+"):
-            concat([df, df[0]], axis=1)
-
 class TestOrderedMerge(tm.TestCase):
 
     def setUp(self):