Repository: pandas-dev/pandas

Issue:
pandas.concat doesn't accept a deque
The `pandas.concat` routine's behaviour is slightly at odds with its documentation (and error messages). The documentation states that the first parameter (objs) accepts "list or dict of Series, DataFrame, ..." but the routine is rather more forgiving and appears to accept lists, tuples, dicts, and generator objects (very useful!); hence my impression (during usage) was that it accepted "iterables" generally. Unfortunately it turns out this isn't the case; attempting to concatenate a deque of DataFrame objects results in the following:

``` python
from collections import deque
import pandas as pd

df = pd.DataFrame.from_dict({'a': [1, 2, 3], 'b': [4, 5, 6]})
d = deque((df, df, df))
pd.concat(d)
```

```
----> 1 pd.concat(d)

/home/dave/arcticenv/lib/python3.4/site-packages/pandas/tools/merge.py in concat(objs, axis, join, join_axes, ignore_index, keys, levels, names, verify_integrity, copy)
    720                        keys=keys, levels=levels, names=names,
    721                        verify_integrity=verify_integrity,
--> 722                        copy=copy)
    723     return op.get_result()
    724 

/home/dave/arcticenv/lib/python3.4/site-packages/pandas/tools/merge.py in __init__(self, objs, axis, join, join_axes, keys, levels, names, ignore_index, verify_integrity, copy)
    735             raise TypeError('first argument must be a list-like of pandas '
    736                             'objects, you passed an object of type '
--> 737                             '"{0}"'.format(type(objs).__name__))
    738 
    739         if join == 'outer':

TypeError: first argument must be a list-like of pandas objects, you passed an object of type "deque"
```

The error message states that the first argument "must be a list-like of pandas objects" (which is slightly different to the documentation, and closer to the actual implementation's behaviour). Given that a deque is iterable but not indexable (similar to a generator expression) it seems to fulfil the criteria of being "list-like".

Digging into the implementation, the test that's failing appears to be the first line of `_Concatenator.__init__` in `pandas.tools.merge` which reads as follows (in my installation):

``` python
        if not isinstance(objs, (list,tuple,types.GeneratorType,dict,TextFileReader)):
            raise TypeError('first argument must be a list-like of pandas '
                            'objects, you passed an object of type '
                            '"{0}"'.format(type(objs).__name__))
```

So it appears the actual set of iterables accepted by `pandas.concat` is lists, tuples, generator expressions, dicts, and instances of `TextFileReader`. I suggest that it might be better to check for (and act upon) special cases and otherwise assume that `objs` is a suitable iterable of `DataFrame` objects. In other words, get rid of that check entirely, and add a couple of checks for "expected" special cases (such as a user mistakenly passing a `DataFrame` as objs; there's already a check in place for dicts a bit further on).

The conversion of `objs` to a list-comprehension later on (below `if keys is None`) should raise a `TypeError` in the case that it isn't iterable so the change shouldn't cause much impact (i.e. in the case of a non-iterable passed as `objs`, it'll raise an exception of the same class as the existing code).

If this sounds reasonable, I'm happy to provide a pull request?



Discussion:
happy to have a pull request to do this
the issue is that list-like can be satisfied by a single pandas object (eg series/DataFrame) and then doesn't give a sensible error message 

so I guess the true meaning here is an iterable that may or may not be indexable (eg dict) that is not a pandas object might work (and maybe need to add generator too as it's not technically in that definition)

doc string improvement also useful

pls add your example as a test but can't change any existing tests


Resolving patch:
diff --git a/pandas/tools/merge.py b/pandas/tools/merge.py
--- a/pandas/tools/merge.py
+++ b/pandas/tools/merge.py
@@ -675,7 +675,7 @@ def concat(objs, axis=0, join='outer', join_axes=None, ignore_index=False,
 
     Parameters
     ----------
-    objs : list or dict of Series, DataFrame, or Panel objects
+    objs : a sequence or mapping of Series, DataFrame, or Panel objects
         If a dict is passed, the sorted keys will be used as the `keys`
         argument, unless it is passed, in which case the values will be
         selected (see below). Any None objects will be dropped silently unless
@@ -731,8 +731,8 @@ class _Concatenator(object):
     def __init__(self, objs, axis=0, join='outer', join_axes=None,
                  keys=None, levels=None, names=None,
                  ignore_index=False, verify_integrity=False, copy=True):
-        if not isinstance(objs, (list,tuple,types.GeneratorType,dict,TextFileReader)):
-            raise TypeError('first argument must be a list-like of pandas '
+        if isinstance(objs, (NDFrame, compat.string_types)):
+            raise TypeError('first argument must be an iterable of pandas '
                             'objects, you passed an object of type '
                             '"{0}"'.format(type(objs).__name__))
 


Test patch:
diff --git a/pandas/tools/tests/test_merge.py b/pandas/tools/tests/test_merge.py
--- a/pandas/tools/tests/test_merge.py
+++ b/pandas/tools/tests/test_merge.py
@@ -2203,6 +2203,33 @@ def test_concat_series_axis1_same_names_ignore_index(self):
         result = concat([s1, s2], axis=1, ignore_index=True)
         self.assertTrue(np.array_equal(result.columns, [0, 1]))
 
+    def test_concat_iterables(self):
+        from collections import deque, Iterable
+
+        # GH8645 check concat works with tuples, list, generators, and weird
+        # stuff like deque and custom iterables
+        df1 = DataFrame([1, 2, 3])
+        df2 = DataFrame([4, 5, 6])
+        expected = DataFrame([1, 2, 3, 4, 5, 6])
+        assert_frame_equal(pd.concat((df1, df2), ignore_index=True), expected)
+        assert_frame_equal(pd.concat([df1, df2], ignore_index=True), expected)
+        assert_frame_equal(pd.concat((df for df in (df1, df2)), ignore_index=True), expected)
+        assert_frame_equal(pd.concat(deque((df1, df2)), ignore_index=True), expected)
+        class CustomIterator1(object):
+            def __len__(self):
+                return 2
+            def __getitem__(self, index):
+                try:
+                    return {0: df1, 1: df2}[index]
+                except KeyError:
+                    raise IndexError
+        assert_frame_equal(pd.concat(CustomIterator1(), ignore_index=True), expected)
+        class CustomIterator2(Iterable):
+            def __iter__(self):
+                yield df1
+                yield df2
+        assert_frame_equal(pd.concat(CustomIterator2(), ignore_index=True), expected)
+
     def test_concat_invalid(self):
 
         # trying to concat a ndframe with a non-ndframe