BUG: Support to create DataFrame from list subclasses
State: closed

- [x] closes #21226
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Alternative fix could be that instead of doing `isinstance` check in frame's `_to_arrays` method:

```
if isinstance(data[0], (list, tuple)):
```

We would do:

```
if type(data[0]) in (list, tuple):
```

This would then assure that `to_object_array` is called really just with exactly lists. But currently there is a mismatch, `if` checks with subtypes, while `to_object_array` does not support them. I think it is better to support them so this merge request adds support for subtypes/subclasses of list.

Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
Hello @mitar! Thanks for updating the PR.





Cheers ! There are no PEP8 issues in this Pull Request. :beers: 

##### Comment last updated on May 29, 2018 at 04:52 Hours UTC
I also ran benchmarks:

```
     [1c2844ac]       [3a64f848]
+         255±1μs         99.7±1ms   391.76  indexing.NumericSeriesIndexing.time_loc_scalar(<class 'pandas.core.indexes.numeric.Float64Index'>)
+        2.51±0ms           5.68ms     2.26  index_object.Ops.time_multiply('float')
+           1.39s            2.45s     1.76  plotting.Misc.time_plot_andrews_curves
+        31.3±1ms       43.5±0.1ms     1.39  join_merge.MergeAsof.time_on_int
+      39.0±0.8ms         52.9±2ms     1.36  eval.Eval.time_mult('python', 1)
+        669±30μs         849±30μs     1.27  join_merge.JoinNonUnique.time_join_non_unique_equal
+         137±5ms          174±2ms     1.27  eval.Eval.time_chained_cmp('python', 'all')
+      35.3±0.7ms       43.6±0.4ms     1.24  categoricals.ValueCounts.time_value_counts(False)
+      57.1±0.5ms         70.3±1ms     1.23  join_merge.MergeAsof.time_by_int
+     4.39±0.05ms      5.40±0.03ms     1.23  rolling.Quantile.time_quantile('Series', 1000, 'int', 1, 'linear')
+           4.80s            5.79s     1.21  gil.ParallelGroups.time_get_groups(8)
+     64.4±0.09ms      75.0±0.08ms     1.16  groupby.Groups.time_series_groups('int64_small')
+         932±2ns      1.08±0.01μs     1.16  timestamp.TimestampProperties.time_quarter(None, None)
+     1.21±0.01μs      1.40±0.03μs     1.15  timestamp.TimestampProperties.time_dayofyear(None, 'B')
+        76.5±2ms         85.9±1ms     1.12  eval.Eval.time_add('numexpr', 'all')
+           280ms            312ms     1.12  gil.ParallelKth.time_kth_smallest
+     3.72±0.01ms         4.14±0ms     1.11  io.csv.ReadCSVFloatPrecision.time_read_csv(';', '.', 'round_trip')
+        3.70±0ms      4.11±0.01ms     1.11  io.csv.ReadCSVFloatPrecision.time_read_csv(',', '_', 'round_trip')
+        4.18±0ms      4.61±0.01ms     1.10  io.csv.ReadCSVFloatPrecision.time_read_csv_python_engine(';', '.', 'high')
+        92.7±3ms        102±0.8ms     1.10  strings.Methods.time_pad
-        66.0±1ms       59.9±0.2ms     0.91  strings.Methods.time_strip
-           103ms       91.1±0.2ms     0.88  frame_methods.Count.time_count_level_mixed_dtypes_multi(0)
-     6.81±0.02ms       5.93±0.1ms     0.87  rolling.Quantile.time_quantile('Series', 1000, 'float', 1, 'midpoint')
-          16.7ms       14.5±0.8ms     0.87  frame_methods.Reindex.time_reindex_upcast
-           1.43s            1.23s     0.86  join_merge.I8Merge.time_i8merge('left')
-         242±1ms          209±3ms     0.86  rolling.VariableWindowMethods.time_rolling('Series', '1d', 'float', 'median')
-     4.44±0.02μs      3.81±0.02μs     0.86  period.PeriodProperties.time_property('min', 'minute')
-          42.2ms           36.1ms     0.85  eval.Query.time_query_datetime_index
-          22.9ms           19.4ms     0.85  eval.Query.time_query_datetime_column
-      2.15±0.3ms      1.77±0.04ms     0.83  frame_methods.Isnull.time_isnull
-     12.1±0.06ms       9.97±0.1ms     0.82  rolling.VariableWindowMethods.time_rolling('Series', '1d', 'int', 'std')
-           908ms            730ms     0.80  join_merge.ConcatPanels.time_c_ordered(2, True)
-      99.1±0.7ms       79.2±0.6ms     0.80  rolling.VariableWindowMethods.time_rolling('Series', '50s', 'int', 'median')
-          15.1ms       12.0±0.2ms     0.80  io.hdf.HDFStoreDataFrame.time_query_store_table
-          10.5ms           8.17ms     0.78  frame_methods.Reindex.time_reindex_axis0
-           1.05s            718ms     0.69  join_merge.ConcatPanels.time_c_ordered(2, False)
-        174±30ms         94.8±2ms     0.55  eval.Eval.time_and('numexpr', 'all')
-        17.2±2ms      8.24±0.07ms     0.48  binary_ops.Ops.time_frame_mult(True, 1)
```

I do not see how any of these changes are related to list constructor, so not sure if they are relevant?
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/21238?src=pr&el=h1) Report
> Merging [#21238](https://codecov.io/gh/pandas-dev/pandas/pull/21238?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/040f06f731a09fc6e0663cada6697f6602b36f1d?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/21238/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/21238?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master   #21238   +/-   ##
=======================================
  Coverage   92.22%   92.22%           
=======================================
  Files         162      162           
  Lines       51828    51828           
=======================================
  Hits        47798    47798           
  Misses       4030     4030
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.62% <ø> (ø)` | :arrow_up: |
| #single | `43% <ø> (ø)` | :arrow_up: |


------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/21238?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/21238?src=pr&el=footer). Last update [040f06f...8661bbb](https://codecov.io/gh/pandas-dev/pandas/pull/21238?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

Can you give a timing of the benchmark that you added (I would just do it with `%timeit` using the same code, that's a bit easier / more reliable) before/after? 
I don't expect a difference, but just to be sure.
I ran the benchmark:

```
· Running 22 total benchmarks (2 commits * 1 environments * 11 benchmarks)
[  0.00%] · For pandas commit hash 403c23d4:
[  0.00%] ·· Building for virtualenv-py3.5-Cython-matplotlib-numexpr-numpy-openpyxl-pytest-scipy-sqlalchemy-tables-xlrd-xlsxwriter-xlwt...
[  0.00%] ·· Benchmarking virtualenv-py3.5-Cython-matplotlib-numexpr-numpy-openpyxl-pytest-scipy-sqlalchemy-tables-xlrd-xlsxwriter-xlwt
[  4.55%] ··· Running frame_ctor.FromDicts.time_list_of_dict                                                                                                                         89.1±0.4ms
[  9.09%] ··· Running frame_ctor.FromDicts.time_nested_dict                                                                                                                             101±1ms
[ 13.64%] ··· Running frame_ctor.FromDicts.time_nested_dict_columns                                                                                                                     103±1ms
[ 18.18%] ··· Running frame_ctor.FromDicts.time_nested_dict_index                                                                                                                    70.4±0.6ms
[ 22.73%] ··· Running frame_ctor.FromDicts.time_nested_dict_index_columns                                                                                                            72.5±0.7ms
[ 27.27%] ··· Running frame_ctor.FromDicts.time_nested_dict_int64                                                                                                                     202±0.2ms
[ 31.82%] ··· Running frame_ctor.FromDictwithTimestamp.time_dict_with_timestamp_offsets                                                                                          79.6±0.2ms;...
[ 36.36%] ··· Running frame_ctor.FromLists.time_frame_from_lists                                                                                                                     60.3±0.4ms
[ 40.91%] ··· Running frame_ctor.FromNDArray.time_frame_from_ndarray                                                                                                                  173±0.3μs
[ 45.45%] ··· Running frame_ctor.FromRecords.time_frame_from_records_generator                                                                                                      195±2ms;...
[ 50.00%] ··· Running frame_ctor.FromSeries.time_mi_series                                                                                                                            209±0.2μs
[ 50.00%] · For pandas commit hash 1c2844ac:
[ 50.00%] ·· Building for virtualenv-py3.5-Cython-matplotlib-numexpr-numpy-openpyxl-pytest-scipy-sqlalchemy-tables-xlrd-xlsxwriter-xlwt...
[ 50.00%] ·· Benchmarking virtualenv-py3.5-Cython-matplotlib-numexpr-numpy-openpyxl-pytest-scipy-sqlalchemy-tables-xlrd-xlsxwriter-xlwt
[ 54.55%] ··· Running frame_ctor.FromDicts.time_list_of_dict                                                                                                                             88.6ms
[ 59.09%] ··· Running frame_ctor.FromDicts.time_nested_dict                                                                                                                             101±1ms
[ 63.64%] ··· Running frame_ctor.FromDicts.time_nested_dict_columns                                                                                                                     104±1ms
[ 68.18%] ··· Running frame_ctor.FromDicts.time_nested_dict_index                                                                                                                    70.7±0.6ms
[ 72.73%] ··· Running frame_ctor.FromDicts.time_nested_dict_index_columns                                                                                                            71.2±0.7ms
[ 77.27%] ··· Running frame_ctor.FromDicts.time_nested_dict_int64                                                                                                                     202±0.3ms
[ 81.82%] ··· Running frame_ctor.FromDictwithTimestamp.time_dict_with_timestamp_offsets                                                                                          76.8±0.3ms;...
[ 86.36%] ··· Running frame_ctor.FromLists.time_frame_from_lists                                                                                                                     59.2±0.6ms
[ 90.91%] ··· Running frame_ctor.FromNDArray.time_frame_from_ndarray                                                                                                                  176±0.2μs
[ 95.45%] ··· Running frame_ctor.FromRecords.time_frame_from_records_generator                                                                                                      203±1ms;...
[100.00%] ··· Running frame_ctor.FromSeries.time_mi_series                                                                                                                            209±0.2μs
BENCHMARKS NOT SIGNIFICANTLY CHANGED.
```
@jorisvandenbossche I am not sure what you are asking. I looks like one run takes around 60 ms from the results above.
What I meant was doing (before and after):

```
In [7]: N = 1000
   ...: M = 100
   ...: data = [[j for j in range(M)] for i in range(N)]

In [8]: %timeit DataFrame(data)
31.1 ms ± 357 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
```

as if you want to time a very specific thing, `%timeit` is both easier and more accurate than ASV in my experience (but asv is still important to keep track of this in long term / benchmark changes that can have wide impact).
But what you posted is fine as well! Performance is not affected.

Oh, I didn't know about `%timeit`.
why is a List subclass actually useful?
I mean, there are different places where you can get data which is a list subclass. So if you want to convert such data to DataFrame, then why not? For example, we are using a list subclass to have additional methods on the list and additional metadata.

In any case, Pandas is currently inconsistent and something has to be changed. Currently is is checking with `isnstance(data, list)` if you can call this method, and then calls it, which fails. So or that check is changed, or this method is changed.

I do not understsand why we would not change this method? There is no performance penalty, and it allows easier interoperability with other data. In general Pandas tries to be well integrated with Python ecosystem, so if something is an instance of list, in Python, this means that you can always use it as list. But Pandas currently does not support this.

So, my question is, why not? What is problematic in this pull request?
I am fine with this patch. It is a minor change and easy to support it. 
In any case, the current error message is not really use friendly (due to the inconsistency @mitar notes between the isinstance check and the actual cython function)
The problem is that, we have many many places where we actually check for lists for various reasons, e.g. its is not a scalar, not a tuple and also we care if things are array-like (IOW have a dtype) or not. So  sure this patch looks ok, but this is actually introducing way more inconsistencies that trying to solve one.
Are you checking elsewhere with `type(...) is` or with `isinstance`?
Any update here? So as I mentioned, I think there is or a problem in code path that it first checks with isinstance and later on requires fixed list type, or that we fix that later on it also checks with isinstance.
if you want to merge master and update, can see where this is
closing as stale. if you'd like to continue, pls ping.
> closing as stale. if you'd like to continue, pls ping.

Hey @jreback I'd like to help close this. Could you reopen the PR please so I commit changes to it?
Hey all, I think the discussion here was already completed and we can merge?
@WillAyd if you have any commets
thanks @mitar 
Thanks @rok.
Thanks all! :)