Repository: pandas-dev/pandas

Issue:
API: Missing support for creating a DataFrame from Python native arrays (or other objects supporting the buffer protocol)
``` python
import pandas
import array

x=array.array('i', range(10))

pdf = pandas.DataFrame.from_items([('a', x)])
# fails with:
# ValueError: If use all scalar values, must pass index
```

Also, since x has a `__len__` may be the error message should not refer to a scalar.

Explicit casting to numpy arrays is a workaround, but it would seem convenient to have it done within the constructor for `DataFrame`:

``` python
import numpy
pdf = pandas.DataFrame.from_items([('a', numpy.asarray(x))])
```



Discussion:
out of curiosity why are you using the python `array` as opposed to a numpy array to begin with?

I am not seeing an opposition in the use of one rather than the other, as I am using using either (or both) depending on the needs.

When I pick Python's array, the reason can be one of:
- Shipping with Python
- Fast

``` bash
$ python -m timeit -s 'import array' -s 'a = array.array("i", range(1000))' -- 'for e in a:' '  pass'
100000 loops, best of 3: 15.8 usec per loop
$ python -m timeit -s 'import numpy' -s 'a = numpy.array(range(1000))' -- 'for e in a:' '  pass'
10000 loops, best of 3: 77.2 usec per loop
```
- Lightweight implementation of the buffer protocol (to try out things)

your first point is valid, though `numpy` is so installed nowadays as its a base for most other packages

you speed test is not testing anything, try doing actual operations

```
In [25]: x=array.array('i', range(100000))

In [28]: %timeit sum(x)
1000 loops, best of 3: 1.09 ms per loop

In [29]: y = np.arange(100000)

In [30]: %timeit y.sum()
10000 loops, best of 3: 72.7 us per loop

```

numpy support the buffer protocol

Note: the issue report is about is about a missing functionality. Should you wish (and have the powers) to decide not to 
implement it, just close the report (grey button). If wishing to tell the world that Python's arrays have no use, the Python mailing lists is where you should head ;-)

Otherwise:
1. The claim that `numpy` is a base for most Python packages is somewhat bold. Data would help believe it.
2. I am comparing the time to access elements in the respective arrays with the exact same code. You are comparing a generic Python function with a specialized C-extension. When outside of the canned C-extensions, Python's arrays are faster.
3. "lightweight" (as in "shipping with Python - no dependency involved")

@lgautier I don't have a problem with your request.

I am simply pointing out to you that IMHO there is no reason to use`array` over `numpy` ever for any kind of actual computation. Yes `array.array` has its uses, but 99% of the time `numpy` or lists are usually the answer.

Your 3rd point is also very odd; if you want lightweight, then how would you expect to use `pandas`, which relies on `numpy`?

your second point is also not a use case. accessing elements is not a relevant benchmark as it tests a very specific criteria (which maybe array is optimized for); `numpy` is optimized for a much broader range of computations.

as to your first point, http://sourceforge.net/projects/numpy/files/NumPy/stats/timeline?dates=2013-01-01+to+2013-07-19, seems like a lot to me

I'd like to keep the issue report on track: this is about suggesting that objects supporting the buffer protocol should be accepted by constructors for `DataFrame` (the Python arrays being the easiest way to have them since the package `array` is part of the Python standard library).

To keep answering to what has moved, in my opinion, to an extraneous discussion, may be because of mutual incomprehension.

Point 1: I looked at the link, and I really could not get the slightest hint that "numpy is a base for most Python packages" (if the thread was more relaxed I could have made the comment that the graph caught my attention first, and that a striking feature is the decreasing number of downloads over time... ;-) ). If the number of download is what I should be looking at, that's not a small number, but a little perspective would not hurt: 

``` bash
$ vanity sqlalchemy | tail -n 1
SQLAlchemy has been downloaded 2183196 times!
$ vanity numpy | tail -n 1
numpy has been downloaded 883206 times!
```

I would certainly not claim that most Python packages are based on SQLAlchemy, or not even that most Python packages having anything to do with SQL are using SQLAlchemy.
My point was that one of the general reasons I can occasionally be using `array`, in answer to your initial question, is that it is shipping with Python (so always there, including alternative Python implementations).

Point 2: accessing elements in an array quickly is either not an operation, or too specialized ?! Come on !
Again, my point here is to give one of the general reasons I can occasionally be using `array`, in answer to your initial question. I said that Python arrays are fast, and backed it up with the exact same Python code on arrays from `numpy` and `array`. I initially did not write /faster/, but you looked for it ;-)

Point 3: that's also in answer to your question about why one would ever use Python's arrays. I just listed what is sometimes making me use `arrays` over `numpy`. `pandas` is requiring `numpy`, but other packages do not necessarily (see point 1 - by the way, did I just see a gimmick "99%" thrown around without backing data ? Tsk, tsk, tsk... not when arguing about scientific computing... ;-) ). The buffer interface is here to allow a loose coupling between packages, and the array package in Python is here to precisely allow that. Should one wish to do operations with `numpy` it can be imported. This issue was unfortunately hijacked, but in the original context this means that `pandas` could be imported whenever required, and the vectors to be bundled in a `DataFrame`.

ok...I have marked this for 0.13. Not a big deal.

Your point 3 is obviously completely False; you cannot use `pandas` without `numpy`. Doesn't matter if it supports fully `array.array` or not. That is irrelevant.

> ok...I have marked this for 0.13.

Thanks.

> ```
>  Not a big deal.
> ```

I don't want to be arguing over a controversial issue then... ;-)

> Your point 3 is obviously completely False; you cannot use pandas without numpy. Doesn't matter if it supports fully array.array or not. That is irrelevant.

Could you point out which part is (allegedly) false ? I'd happy to try clarifying.

Your comment suggests that you might have understood that I am claiming that `pandas` can be used without `numpy`, leaving me quite puzzled since I wrote the exact opposite:

> `pandas` is requiring `numpy`

@lgautier
1. In point 3, you seem to be implying that users should have a choice between using `array` or using `numpy` when using `pandas`. This simply will never happen. Furthermore, `array` instances will be turned into `numpy` arrays anyway, if this supported, so at best it's introducing a redundancy for the very rare case (and no, I don't have any empirical evidence for this, just anecdotal) of using `array` instead of `numpy`. IMHO, It's very silly to use `array` instead of `numpy`. Element access is a weak benchmark, since most of the time you're not using arrays because you want fast element-by-element access, you want to do operations on the _whole_ array so that benchmark isn't very realistic. If you're going to use `pandas` then you need `numpy`, so again, why not just use that?
2. Yes, other packages don't require `numpy`, but so what? `pandas` will probably have `numpy` as a dependency for the foreseeable future.
3. Can you honestly provide a _real-world_ use case where `array` outperforms `numpy`, other than element access?
4. Python's `array`s are a "canned" C-extension:

```
Python 2.7.5 (default, May 17 2013, 07:55:04)
[GCC 4.8.0 20130502 (prerelease)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import array
>>> array
<module 'array' from '/home/phillip/.virtualenvs/pandas/lib/python2.7/lib-dynload/array.so'>
```

> When outside of the canned C-extensions, Python's arrays are faster.

What evidence is there to support this, other than the rarely-used element-by-element access in pure Python? A factor of ~3 for a rare use case is not evidence.

@lgautier 

sorry this got out of hand :)

I was honestly trying to find a usecase for `arrays`, so was interested in why you were using them.

I think supporting objects that implement the buffer protocol is probably okay. E.g., you could pass a Cython `memoryview` array into `DataFrame`. This would be useful if we get the ball rolling on using fused types since we could just return Cython arrays and not have to worry about converting them to numpy arrays in Cython. Although, I'm not sure if any of the Cython routines are used at such a high level. I would imagine they are called by Python functions that `DataFrame` calls and thus it would never even see any Cython arrays

@cpcloud you could easily do that in wrappers, `DataFrame` is too high level for that, and in any event usually need  the 3 different (data,index,columns) arrays to really _do_ anything.

yeah was thinking something similar. crossing too much of the api for something probably not that useful

@cpcloud 

> IMHO, It's very silly to use array instead of numpy. Element access is a weak benchmark, since most of the time you're not using arrays because you want fast element-by-element access, you want to do operations on the whole array so that benchmark isn't very realistic.

<sigh>  now we have a combo: unsubstantiated judgment (first part), lecturing people about what their code is doing (without knowing much of the said code), and comments (there and in the rest of the post) largely showing a miscomprehension of what is discussed.

Sorry for teasing, but can you read the thread ?

@lgautier Why don't you just submit a PR then instead of wasting time on pedantic arguments? It sounds like you're the only one that understands what's going on in your head, so let the code speak for itself.

@jreback 

> sorry this got out of hand :)

I am tempted to say: "I did not expect the Spanish Inquisition"
;-)

> I was honestly trying to find a usecase for arrays, so was interested in why you were using them.

Sure. 

Note that I understand the question as:
"why would one ever use `array` instead of `numpy` ?" 

("where does `array` fit when `pandas` requires `numpy` ?" is for later).

While `numpy` is providing a number of useful operations on arrays (initially matrix multiplication and linear algebra, by wrapping BLAS/LAPACK) there are situations where the most important part is to have an efficient structure for storing fixed-size numerical arrays (lists or tuple use quickly much more memory)... and Python `array` provides just that.

To give a concrete example. I can have a simple module "foo.py"

``` python
from array import array

def load_data():
    genome_pos = array.fromfile('genome_pos.dat') # sorted genomic positions
    signal = array.fromfile('signal.dat') # some signal (let's keep it short, this is an example)
    return (genome_pos, signal)

import bisect
def interval_indices(genome_pos, pos_a, pos_b):
    i_a = bisect.bisect(genome_pos, pos_a)
    i_b = bisect.bisect(genome_pos, pos_b)
    return (i_a, i_b)
```

That module can be used to put together a quick application.

``` python
import foo
genome_pos, signal = foo.load_data()

from flask import Flask
app = Flask(__name__)
from flask import request

@app.route('/slice/<pos_a>/<pos_b>')
def hello(pos_a=None, pos_b=None):
    i_a, i_b = foo.interval_indices(genome_pos, pos_a, pos_b)
    return render_template('xyplot.html', x=genome_pos[i_a:i_b], y=signal[i_a:i_b])

if __name__ == '__main__':
    app.run()
```

`numpy` could be used in place of `array`, but `array` is up to the task and is shipping with Python. I often have to write code to be run elsewhere and I weight every dependency on 3rd-party modules.

This leads to the second question:
"where does `array` fit when `pandas` requires `numpy` ?"

The convenience of the buffer interface is that, should computation with `numpy` (or `OpenCV`... just to be annoying ;-) ) be wished, the underlying data can be wrapped up seamlessly. I can ensure minimal capabilities without requiring `numpy` or else.

The buffer interface turned out to be out of the equation for the fix.

Pull request:
https://github.com/pydata/pandas/pull/4317


Resolving patch:
diff --git a/pandas/core/frame.py b/pandas/core/frame.py
--- a/pandas/core/frame.py
+++ b/pandas/core/frame.py
@@ -16,6 +16,7 @@
 import sys
 import collections
 import warnings
+import types
 
 from numpy import nan as NA
 import numpy as np
@@ -24,7 +25,7 @@
 from pandas.core.common import (isnull, notnull, PandasError, _try_sort,
                                 _default_index, _maybe_upcast, _is_sequence,
                                 _infer_dtype_from_scalar, _values_from_object,
-                                _coerce_to_dtypes, _DATELIKE_DTYPES)
+                                _coerce_to_dtypes, _DATELIKE_DTYPES, is_list_like)
 from pandas.core.generic import NDFrame
 from pandas.core.index import Index, MultiIndex, _ensure_index
 from pandas.core.indexing import (_NDFrameIndexer, _maybe_droplevels,
@@ -413,12 +414,14 @@ def __init__(self, data=None, index=None, columns=None, dtype=None,
             else:
                 mgr = self._init_ndarray(data, index, columns, dtype=dtype,
                                          copy=copy)
-        elif isinstance(data, list):
+        elif isinstance(data, (list, types.GeneratorType)):
+            if isinstance(data, types.GeneratorType):
+                data = list(data)
             if len(data) > 0:
                 if index is None and isinstance(data[0], Series):
                     index = _get_names_from_index(data)
 
-                if isinstance(data[0], (list, tuple, collections.Mapping, Series)):
+                if is_list_like(data[0]) and getattr(data[0],'ndim',0) <= 1:
                     arrays, columns = _to_arrays(data, columns, dtype=dtype)
                     columns = _ensure_index(columns)
 
@@ -4545,7 +4548,7 @@ def isin(self, values, iloc=False):
 
 
         else:
-            if not com.is_list_like(values):
+            if not is_list_like(values):
                 raise TypeError("only list-like or dict-like objects are"
                                 " allowed to be passed to DataFrame.isin(), "
                                 "you passed a "
@@ -4705,7 +4708,7 @@ def extract_index(data):
             elif isinstance(v, dict):
                 have_dicts = True
                 indexes.append(list(v.keys()))
-            elif isinstance(v, (list, tuple, np.ndarray)):
+            elif is_list_like(v) and getattr(v,'ndim',0) <= 1:
                 have_raw_arrays = True
                 raw_lengths.append(len(v))
 


Test patch:
diff --git a/pandas/tests/test_frame.py b/pandas/tests/test_frame.py
--- a/pandas/tests/test_frame.py
+++ b/pandas/tests/test_frame.py
@@ -2606,6 +2606,57 @@ def test_constructor_list_of_lists(self):
         self.assert_(com.is_integer_dtype(df['num']))
         self.assert_(df['str'].dtype == np.object_)
 
+    def test_constructor_sequence_like(self):
+        # GH 3783
+        # collections.Squence like
+        import collections
+
+        class DummyContainer(collections.Sequence):
+            def __init__(self, lst):
+                self._lst = lst
+            def __getitem__(self, n):
+                return self._lst.__getitem__(n)
+            def __len__(self, n):
+                return self._lst.__len__()
+
+        l = [DummyContainer([1, 'a']), DummyContainer([2, 'b'])]
+        columns = ["num", "str"]
+        result = DataFrame(l, columns=columns)
+        expected = DataFrame([[1,'a'],[2,'b']],columns=columns)
+        assert_frame_equal(result, expected, check_dtype=False)
+
+        # GH 4297
+        # support Array
+        import array
+        result = DataFrame.from_items([('A', array.array('i', range(10)))])
+        expected = DataFrame({ 'A' : list(range(10)) })
+        assert_frame_equal(result, expected, check_dtype=False)
+
+        expected = DataFrame([ list(range(10)), list(range(10)) ])
+        result = DataFrame([ array.array('i', range(10)), array.array('i',range(10)) ])
+        assert_frame_equal(result, expected, check_dtype=False)
+
+    def test_constructor_iterator(self):
+
+        expected = DataFrame([ list(range(10)), list(range(10)) ])
+        result = DataFrame([ range(10), range(10) ])
+        assert_frame_equal(result, expected)
+
+    def test_constructor_generator(self):
+        #related #2305
+
+        gen1 = (i for i in range(10))
+        gen2 = (i for i in range(10))
+
+        expected = DataFrame([ list(range(10)), list(range(10)) ])
+        result = DataFrame([ gen1, gen2 ])
+        assert_frame_equal(result, expected)
+
+        gen = ([ i, 'a'] for i in range(10))
+        result = DataFrame(gen)
+        expected = DataFrame({ 0 : range(10), 1 : 'a' })
+        assert_frame_equal(result, expected)
+
     def test_constructor_list_of_dicts(self):
         data = [OrderedDict([['a', 1.5], ['b', 3], ['c', 4], ['d', 6]]),
                 OrderedDict([['a', 1.5], ['b', 3], ['d', 6]]),
diff --git a/pandas/tests/test_series.py b/pandas/tests/test_series.py
--- a/pandas/tests/test_series.py
+++ b/pandas/tests/test_series.py
@@ -353,6 +353,12 @@ def test_constructor_series(self):
 
         assert_series_equal(s2, s1.sort_index())
 
+    def test_constructor_iterator(self):
+
+        expected = Series(list(range(10)))
+        result = Series(range(10))
+        assert_series_equal(result, expected)
+
     def test_constructor_generator(self):
         gen = (i for i in range(10))